Optimization for Medical Image Segmentation: Theory and Practice when evaluating with Dice Score or Jaccard Index

Tom Eelbode; Jeroen Bertels; Maxim Berman; Dirk Vandermeulen; Frederik Maes; Raf Bisschops; Matthew B. Blaschko

Source-linked AI summary

Optimization for Medical Image Segmentation: Theory and Practice when evaluating with Dice Score or Jaccard Index

Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, Matthew B. Blaschko

arXiv:2010.13499v1Oct 26, 2020eess.IVcs.CVcs.LG

TL;DR

The paper addresses the mismatch between Dice/Jaccard evaluation and frequent training with per-pixel cross-entropy losses. It analyzes metric-sensitive surrogates and weighting schemes theoretically, then validates them across medical segmentation settings. Metric-sensitive losses outperform cross-entropy-based losses for Dice and Jaccard evaluation, supporting their broader use within this scope.

Problem
Many segmentation studies evaluate with Dice or Jaccard but train with losses that do not directly optimize those metrics.
Method
The paper combines theoretical analysis of Dice, Jaccard, weighted Hamming, and Tversky with empirical validation across six binary medical segmentation tasks and additional settings.
Results
Metric-sensitive losses outperform cross-entropy-based losses for Dice and Jaccard evaluation, while the evaluated Dice and Jaccard surrogates show no statistically significant differences.
Takeaways & Limitations
Metric-sensitive losses are preferable when segmentation performance is evaluated with Dice or Jaccard.

Abstract

from arXiv · show

In many medical imaging and classical computer vision tasks, the Dice score and Jaccard index are used to evaluate the segmentation performance. Despite the existence and great empirical success of metric-sensitive losses, i.e. relaxations of these metrics such as soft Dice, soft Jaccard and Lovasz-Softmax, many researchers still use per-pixel losses, such as (weighted) cross-entropy to train CNNs for segmentation. Therefore, the target metric is in many cases not directly optimized. We investigate from a theoretical perspective, the relation within the group of metric-sensitive loss functions and question the existence of an optimal weighting scheme for weighted cross-entropy to optimize the Dice score and Jaccard index at test time. We find that the Dice score and Jaccard index approximate each other relatively and absolutely, but we find no such approximation for a weighted Hamming similarity. For the Tversky loss, the approximation gets monotonically worse when deviating from the trivial weight setting where soft Tversky equals soft Dice. We verify these results empirically in an extensive validation on six medical segmentation tasks and can confirm that metric-sensitive losses are superior to cross-entropy based loss functions in case of evaluation with Dice Score or Jaccard Index. This further holds in a multi-class setting, and across different object sizes and foreground/background ratios. These results encourage a wider adoption of metric-sensitive loss functions for medical segmentation tasks where the performance measure of interest is the Dice score or Jaccard index.

I. INTRODUCTION

Dice and Jaccard are widely used because they better reflect segmentation quality than pixel-wise accuracy, yet many studies train with losses that do not directly optimize them. The paper analyzes these choices theoretically and empirically, finding metric-sensitive surrogates preferable to cross-entropy-based losses.

Motivation: Dice and Jaccard are commonly used because they better indicate perceptual segmentation quality than pixel-wise accuracy.Dice also better reflects object size and localization agreement.
Motivation: In MICCAI 2018, 68 of 96 learning-based segmentation papers used losses that did not directly optimize Dice or Jaccard despite evaluating with those metrics.Pixel-wise weighted cross-entropy remained frequent, while soft Dice, soft Jaccard, and Lovász-softmax provide differentiable alternatives.
Contributions: The paper theoretically studies relations among Dice, Jaccard, weighted Hamming, and Tversky similarities under risk minimization.It asks whether weighted cross-entropy or Tversky weighting can surrogate Dice or Jaccard.
Contributions: Dice and Jaccard approximate each other, whereas no analogous approximation is established for weighted Hamming similarity.The Tversky index specializes to Dice at α = β = 1/2 and to Jaccard at α = β = 1.
Empirical findings: Cross-entropy performs worse than every evaluated Dice or Jaccard surrogate, while the surrogates do not differ significantly from one another.The extended work adds datasets, architectures, Tversky analysis, variant comparisons, and multi-class and object-size experiments.

A. Differentiable loss surrogates

Because neural-network outputs are continuous while segmentation similarities are defined on binary masks, differentiable surrogate losses are used for gradient-based training. The section describes soft Dice, soft Jaccard, Lovász-based, Tversky, and weighted cross-entropy formulations.

Surrogate construction: Discrete segmentation similarities are relaxed to continuous predictions in [0, 1]^d so gradient descent can optimize them.Softmax can constrain unbounded scores to this interval.
Other surrogates: Weighted cross-entropy can optimize weighted Hamming similarity, while soft Tversky introduces explicit false-positive and false-negative weights.The section compares these alternatives with metric-sensitive surrogates for Dice and Jaccard.
Soft Dice: Soft Dice generalizes Dice by replacing set intersections and cardinalities with inner products and relaxing predicted labels to [0, 1]^d.Alternative relaxations can replace the squared L2 term with an Lp norm, changing statistical and optimization properties.
Soft Jaccard: Soft Jaccard uses an analogous relaxation, with a formulation based on the inner product and L1 norms of the ground-truth and predicted masks.The displayed loss is ΔJ(y, ỹ) = 1 − ⟨y, ỹ⟩/(∥y∥1 + ∥ỹ∥1 − ⟨y, ỹ⟩).
Lovász-softmax: Lovász-softmax constructs convex, almost-everywhere differentiable surrogates for submodular losses, including Jaccard but not Dice.It projects pixel scores into [0, 1] and computes the Lovász extension.

B. Approximation bounds

Dice and Jaccard approximate each other with explicit relative and absolute error bounds, whereas weighted Hamming similarity does not provide a nontrivial approximation. Tversky’s approximation worsens as its parameters move away from α = β = 0.5, while cross-entropy offers no Dice or Jaccard approximation guarantee.

Dice and Jaccard approximate each other with relative error 1 and absolute error 3−2√2 = 0.17157...
Dice and weighted Hamming similarity do not relatively approximate each other and have only the trivial absolute error bound of 1.
Dice and unweighted Hamming similarity likewise do not relatively approximate each other and absolutely approximate each other with error 1.
Jaccard loss provides multiplicative and additive approximation guarantees for Dice, and Jaccard-risk optimization minimizes an upper bound on Dice risk.
Tversky approximates Dice with explicit absolute and relative error bounds that increase as α and β deviate from 0.5.
Optimal Dice performance is achieved at α = β = 0.5, while cross-entropy and weighted cross-entropy can yield arbitrarily poor Dice results.

III. EMPIRICAL SETUP

The empirical study compares six segmentation losses across six binary medical image segmentation tasks, including public and in-house datasets with varied dimensionality and object characteristics.

Six losses are evaluated: CE, wCE, sDice, sJaccard, Lovász-sigmoid, and sTversky.
The six binary tasks cover brain tumors, ischemic stroke lesions, ischemic core, dental molars, colorectal polyps, and white matter hyperintensities.
The datasets include four public 3D datasets and two in-house 2D datasets.
The study additionally evaluates multi-class glioma-subregion segmentation on BRATS 2018 using whole tumor, tumor core, and enhancing tumor volumes.

A. Data preprocessing and network architectures

The study uses U-Net-like architectures adapted to dataset dimensionality and preprocesses inputs through modality selection, resampling, resizing, cropping, and patch-wise processing where needed.

A 3D U-Net-like architecture is used for BR18, IS17, IS18, and WM17, while MO17 uses the same design with 2D convolutions.
The architecture is based on No New-Net, with one fewer level and modified pooling and upsampling to preserve field of view.
All available image modalities are used except perfusion data for IS17 and IS18, and public datasets are resampled to 2 mm isotropic voxels.
Inputs are resized and cropped for memory constraints, with patch-wise processing used for WM17.

B. Training parameters

Training begins with cross-entropy pretraining and then compares several loss functions under dataset-specific optimization settings, using heuristic class balancing for weighted cross-entropy.

CNNs are pretrained with CE to support final convergence and improve robustness of metric-sensitive losses.
U-Net and DeepLab use Adam, while DeepMedic uses SGD with learning-rate reduction triggered by validation-loss stagnation.
After CE convergence, training continues with CE, wCE, sDice, sJaccard, Lovász, or Tversky using a reset optimizer state.
Weighted cross-entropy uses a foreground-class weight of 1/(2p) and balances foreground and background frequencies heuristically.

C. Model choice and statistical testing

The study uses cross-validation, validation-loss model selection, pairwise loss comparisons, and non-parametric bootstrap tests to evaluate segmentation performance.

Five-fold cross-validation aggregates metrics over all left-out subjects.
Models are selected using the best validation loss, evaluated on full images for patch-wise training.
Losses are compared pairwise, with statistical significance assessed through non-parametric bootstrapping.

IV. RESULTS AND DISCUSSION

The results section compares loss families, soft Dice variants, and experimental settings. It emphasizes how surrogate choice and optimization geometry relate to Dice and Jaccard performance.

Experimental scope: The evaluation covers binary and multi-class segmentation, class imbalance, and the influence of object size.
Loss-function groups: The study distinguishes CE-based losses from metric-sensitive surrogates for Dice, Jaccard, Lovász, and Tversky indices.
Soft Dice variants: L1- and L2-based soft Dice losses differ in optimization properties, with L2 variants flatter around the minimum.
Soft Dice variants: The L1-based soft Dice surrogate shows a favorable performance trend and significantly outperforms the L2 version on IS18, PO18, and WM17DM.
Soft Dice variants: Subsequent experiments restrict analysis to the L1-based relaxation, called the soft Dice loss.

B. Comparison of losses for binary segmentation

Metric-sensitive losses generally outperform cross-entropy-based losses on Dice and Jaccard evaluation, while Dice and Jaccard produce consistent loss rankings. Equal Tversky weighting is the strongest tested setting, whereas weighted cross-entropy is difficult to tune across applications.

Metric agreement: Dice score and Jaccard index provide equivalent loss rankings in theory and empirically.
CE versus metric-sensitive losses: Metric-sensitive losses clearly provide superior Dice scores and Jaccard indexes to CE-based losses, except on MO17 for CE.
Tversky weighting: For Tversky loss, α = β = 0.5, equivalent to Dice loss, is the optimal weighting in all experiments evaluated by Dice score.
Tversky weighting: Tversky performance drops significantly when weighting moves sufficiently far from the α = 0.5 baseline.
Cross-entropy weighting: Weighted cross-entropy typically underperforms cross-entropy, and its useful object-size range is narrower.
Cross-entropy weighting: A suitable weighted cross-entropy weight is application-dependent and may require additional parameter tuning.

4) Qualitative inspection:

Qualitative and stratified analyses show that metric-sensitive losses produce more reliable segmentations than cross-entropy-based losses across object sizes and foreground/background ratios. The additional metric analysis indicates that this advantage is specific to Dice- and Jaccard-oriented evaluation rather than universal across metrics.

Qualitative inspection: Metric-sensitive predictions had narrower output ranges near 1.0, whereas CE outputs varied more and systematically under-segmented structures at a 0.5 threshold.After thresholding, sDice and sJaccard delineations were highly similar; Lovász predictions were similar except for value ranges in the ISLES datasets.
Alternative metrics: For accuracy, CE was often superior, whereas sDice was never inferior to CE for Hausdorff distance and absolute volume difference and was sometimes significantly better.This pattern is consistent with CE directly optimizing accuracy, while metric-sensitive losses target Dice and Jaccard.
Object size: Metric-sensitive losses outperformed CE across nearly the entire object-size range, while wCE improved only for some size intervals and could perform substantially worse.The size-dependent behavior of wCE did not provide a surrogate for the target metric across all scales.

D. Comparison of losses for multi-class segmentation

In multi-class BRATS segmentation, metric-sensitive losses consistently achieved higher Dice scores and Jaccard indexes than cross-entropy losses across tumor regions. Tversky experiments further identified the Dice-equivalent weighting as the best-supported setting for Dice evaluation.

Multi-class comparison: Metric-sensitive losses obtained higher Dice scores and Jaccard indexes than cross-entropy losses for whole tumor, tumor core, and enhancing tumor regions.The same trend appeared across foreground/background ratios of 0.065, 0.028, and 0.012 for WT, TC, and ET, respectively.
Scope: The study extends metric-sensitive-loss analysis beyond binary segmentation to multi-class BRATS evaluation.The broader analysis also considers the growing use of metric-sensitive losses in medical imaging and computer vision.
Tversky weighting: No Tversky weighting scheme was significantly superior to the Dice-equivalent setting α = β = 0.5.Deviations from 0.5 in either weight showed a clear trend toward sub-optimal performance.

APPENDIX A MICCAI 2018 PROCEEDINGS

The appendix documents the paper’s empirical scope, supplementary analyses, and visualization conventions. It covers loss-function usage in MICCAI 2018 and additional robustness analyses across datasets, metrics, class structures, and Tversky weightings.

MICCAI 2018 review: Among 96 learning-based MICCAI 2018 segmentation papers, the review found that evaluation commonly used Dice or Jaccard despite frequent use of non-metric-sensitive losses.The appendix categorizes the proceedings using the flowchart in Figure 6.
F measures: For Dice evaluation, α = β = 0.5 is the optimal Tversky weighting, while alternative weightings can improve selected F-measures.Higher α favors lower-order F-measures, whereas F2.0 performs best in a lower α range; no clear difference is observed among several balanced settings.
Variability: Supplementary boxplots show the greatest variability for IS17, IS18, and PO18, consistent with their difficult nature and smaller sample sizes.These plots provide spread information for the measurements reported in the main tables.
Supplementary analyses: Appendix figures provide boxplots for Dice, Jaccard, accuracy, Hausdorff distance, absolute volume difference, and alternative F-measures across loss functions and datasets.The supplementary plots include soft-Dice variants, virtual PO18 datasets, multi-class BRATS, and Tversky-weight ranges.