Source-linked AI summary
Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory & Practice
Jeroen Bertels, Tom Eelbode, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, Matthew Blaschko
TL;DR
The paper asks whether commonly used cross-entropy training can be principled as a surrogate for Dice or Jaccard evaluation. It analyzes approximation relationships theoretically and compares loss functions across five medical segmentation tasks, finding metric-sensitive surrogates outperform cross-entropy variants while generally showing no statistical differences among themselves.
Problem
Cross-entropy-based training is often used despite evaluation with Dice or Jaccard, raising whether a principled weighting can align cross-entropy with these target metrics.
Method
The paper derives risk-minimization approximation bounds and evaluates cross-entropy, weighted cross-entropy, soft Dice, soft Jaccard, and Lovász losses across five medical segmentation tasks.
Results
Metric-sensitive losses generally achieve higher Dice and Jaccard scores than cross-entropy variants, while their rankings and performance are generally not statistically different from one another.
Takeaways & Limitations
The findings support wider adoption of metric-sensitive Dice- and Jaccard-based losses instead of cross-entropy when these metrics evaluate segmentation.
Abstract
from arXiv · showhide
The Dice score and Jaccard index are commonly used metrics for the evaluation of segmentation tasks in medical imaging. Convolutional neural networks trained for image segmentation tasks are usually optimized for (weighted) cross-entropy. This introduces an adverse discrepancy between the learning optimization objective (the loss) and the end target metric. Recent works in computer vision have proposed soft surrogates to alleviate this discrepancy and directly optimize the desired metric, either through relaxations (soft-Dice, soft-Jaccard) or submodular optimization (Lovász-softmax). The aim of this study is two-fold. First, we investigate the theoretical differences in a risk minimization framework and question the existence of a weighted cross-entropy loss with weights theoretically optimized to surrogate Dice or Jaccard. Second, we empirically investigate the behavior of the aforementioned loss functions w.r.t. evaluation with Dice score and Jaccard index on five medical segmentation tasks. Through the application of relative approximation bounds, we show that all surrogates are equivalent up to a multiplicative factor, and that no optimal weighting of cross-entropy exists to approximate Dice or Jaccard measures. We validate these findings empirically and show that, while it is important to opt for one of the target metric surrogates rather than a cross-entropy-based loss, the choice of the surrogate does not make a statistical difference on a wide range of medical segmentation tasks.
1 Introduction
Dice and Jaccard are widely used because they better reflect segmentation overlap and perceptual quality than pixel-wise accuracy. The paper examines whether training losses can be theoretically aligned with these metrics and validates the comparison empirically.
- Dice and Jaccard are among the most popular performance metrics for medical image segmentation.
- Dice reflects both size and localization agreement, aligning more closely with perceptual quality than pixel-wise accuracy.
- Risk minimization motivates differentiable surrogates such as soft Dice, soft Jaccard, and Lovász-softmax for gradient-based training.
- Despite metric-based evaluation, pixel-wise cross-entropy and weighted cross-entropy remain widely used for training segmentation models.
- The paper finds metric-sensitive losses outperform cross-entropy variants empirically, while generally differing little from one another statistically.
2 Risk minimization with Dice and related similarities
The paper formulates segmentation training as risk minimization over set-based similarities and studies approximation relationships among Dice, Jaccard, Hamming, and weighted Hamming losses. Dice and Jaccard mutually approximate each other, whereas weighted Hamming cannot provide a relative approximation to Dice.
- Empirical risk minimization learns a mapping from observed input x to hidden variable y by minimizing expected loss over a finite training set.
- Binary segmentation represents foreground labels as sets of pixels, enabling set-theoretic similarities including Dice, Jaccard, Hamming, and weighted Hamming.
- Jaccard and Dice are linked by J(y, ỹ) = D(y, ỹ)/(2 − D(y, ỹ)) and D(y, ỹ) = 2J(y, ỹ)/(1 + J(y, ỹ)).
- Proposition 1 states that Jaccard and Dice mutually approximate with relative error 1 and absolute error 3 − 2√2 = 0.17157….
- Proposition 2 states that Dice and optimally weighted Hamming do not relatively approximate each other, with only the trivial absolute error bound of 1.
- A weighted binary loss can be arbitrarily poor for Dice on small objects, whereas Jaccard provides multiplicative and additive approximation guarantees.
3 Empirical setup
The empirical study compares five segmentation losses across five medical binary segmentation tasks using cross-validation. It includes cross-entropy variants, metric-sensitive surrogates, and datasets spanning public 3D and in-house 2D imaging.
- The study compares cross-entropy, weighted cross-entropy, soft Dice, soft Jaccard, and Lovász-sigmoid.
- Evaluation uses cross-validation on five medical binary segmentation tasks.
- The benchmark includes BRATS 2018, ISLES 2017, ISLES 2018, lower-left third molar radiographs, and colorectal polyp images.
4 Results and discussion
Across five-fold evaluations, metric-sensitive losses generally outperform cross-entropy losses on Dice and Jaccard, while metric-sensitive surrogates show no significant differences within their group. These findings extend beyond small objects and across most foreground-area ranges.
- Experimental comparison: Five-fold cross-validation compares CE, weighted CE, soft Dice, soft Jaccard, and Lovász losses using Dice and Jaccard across medical segmentation datasets.Table 1 reports dataset-level scores and significance comparisons for the two loss groups.
- Performance of the surrogates: Metric-sensitive losses generally achieve higher Dice and Jaccard scores than CE and weighted CE.The reported results identify CE and weighted CE as inferior to metric-sensitive losses, with MO17 as an exception for CE.
- Performance of the surrogates: No statistically significant performance difference generally appears among soft Dice, soft Jaccard, and Lovász losses.The result leaves the choice among metric-sensitive losses open for the evaluated tasks.
- Foreground-area dependence: Metric-sensitive losses perform as well as or better than cross-entropy over most relative foreground-area ranges.Figure 1 averages Dice scores within ten equal-sized dataset regions; ISLES 2017 is omitted because of its lower sample count.
- Foreground-area dependence: Cross-entropy can perform poorly across nearly all object scales, while weighted cross-entropy is not consistently sufficient across datasets and area ranges.The strongest examples are BR18 and IS18 for CE, and BR18 and MO17 for weighted CE.
5 Conclusion
The paper combines theoretical analysis with experiments on five medical segmentation tasks to compare Dice-, Jaccard-, Lovász-, and cross-entropy-based optimization. Dice and Jaccard approximate each other, whereas weighted cross-entropy does not provide such an approximation, and metric-sensitive losses are generally preferable in evaluation by these metrics.
- Conclusion: Dice and Jaccard approximate each other under both relative and absolute approximation bounds.The conclusion presents this equivalence as a theoretical result supported by the empirical study.
- Conclusion: No approximation by weighted Hamming similarity, the set-theoretic counterpart of weighted cross-entropy, can be found.This distinguishes weighted cross-entropy from the relationship established between Dice and Jaccard.
- Conclusion: Experiments on five medical segmentation tasks confirm the theoretical findings.The study compares five loss functions from theoretical and empirical perspectives.
- Conclusion: Metric-sensitive losses generally show no significant differences among themselves, while cross-entropy and weighted cross-entropy are inferior on Dice and Jaccard.The conclusion frames this pattern as consistent with the theory that Jaccard controls Dice loss.
- Conclusion: The authors suggest wider adoption of metric-sensitive losses such as Dice and Jaccard.This recommendation follows the reported theory and empirical results within the evaluated scope.