PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Temperature Scaling for Probability Calibration

Zhipeng Ding, Xu Han, Peirong Liu, Marc Niethammer

arXiv:2008.05105v2cs.CV

TL;DR

Semantic segmentation probabilities are often uncalibrated, while existing calibration approaches largely target classification. The paper proposes local temperature scaling, which predicts spatially varying temperatures as post-processing while preserving segmentation accuracy. Across COCO, CamVid, and LPBA40, it reports better calibration than approaches unable to account for spatially varying miscalibration, and improved joint-label-fusion segmentations.

  • Problem

    Semantic segmentation networks can be overconfident, and calibration approaches designed mainly for classification have limited fit for multi-label segmentation and spatially varying miscalibration.

    Source: §1 · p. 1

  • Method

    The paper uses a learning-based local temperature-scaling method to calibrate a segmentation network's probabilities as accuracy-preserving post-processing.

    Source: §1 · pp. 1–2

  • Results

    Across COCO, CamVid, and LPBA40, LTS outperforms calibration approaches that cannot account for spatially varying miscalibration, and joint label fusion improves when combined with probability calibration.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Because LTS preserves segmentation accuracy, it can serve as a post-processing calibration step for existing segmentation models.

    Source: §1 · pp. 1–2

    Pixel- or voxel-level annotation can be difficult, especially at object boundaries, and outliers can produce MCE values near 100% for bin-based evaluation.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

For semantic segmentation, label probabilities are often uncalibrated as they are typically only the by-product of a segmentation task. Intersection over Union (IoU) and Dice score are often used as criteria for segmentation success, while metrics related to label probabilities are not often explored. However, probability calibration approaches have been studied, which match probability outputs with experimentally observed errors. These approaches mainly focus on classification tasks, but not on semantic segmentation. Thus, we propose a learning-based calibration method that focuses on multi-label semantic segmentation. Specifically, we adopt a convolutional neural network to predict local temperature values for probability calibration. One advantage of our approach is that it does not change prediction accuracy, hence allowing for calibration as a post-processing step. Experiments on the COCO, CamVid, and LPBA40 datasets demonstrate improved calibration performance for a range of different metrics. We also demonstrate the good performance of our method for multi-atlas brain segmentation from magnetic resonance images.

1. Introduction

Semantic segmentation networks need reliable confidence estimates alongside accurate labels, yet they tend to be overconfident and existing calibration methods have limited fit for multi-label segmentation. The paper proposes local, post-processing temperature scaling that preserves label order and segmentation accuracy.

1. Introduction

  • Semantic segmentation networks should indicate when predictions may be incorrect, including in autonomous-driving and brain-tumor settings.The introduction emphasizes that accurate labels alone are insufficient without confidence measures.Source: §1 · p. 1
  • CNN segmentation models tend to be overconfident, while existing calibration approaches are mainly designed for classification and have limitations for segmentation.The paper notes challenges with pixel- or voxel-level annotation, model-specific approaches, and one-vs-rest decomposition that may alter accuracy.Source: §1 · p. 1
  • Global temperature scaling cannot capture spatially varying miscalibration, motivating an approach adaptive to local image changes.The paper's goal is post-processing calibration for multi-label segmentation that retains probability order and prediction accuracy.Source: §1 · p. 1
  • The proposed learning-based local temperature scaling predicts a separate temperature for each pixel or voxel, while post-processing leaves segmentation accuracy unchanged.The method is intended for multi-label semantic segmentation and retains label probability order.Source: §1 · pp. 1–2

2. Related Work

The related work covers calibration approaches that do not address spatial variation in semantic segmentation, and methods that improve calibration only indirectly. The paper positions its approach as post-processing that accounts for pixel or voxel correlations.

2. Related Work

  • Bin-based calibration methods such as histogram binning, isotonic regression, and ENIR do not model neighboring-pixel or voxel correlations, whereas the proposed method uses convolutional filters.These methods are described as non-parametric approaches.Source: §2 · p. 2
  • Existing temperature-scaling variants address other calibration settings, but the cited approaches do not account for spatial variation in probability calibration.The discussion includes ATS and BTS alongside Platt, matrix, vector, and standard temperature scaling.Source: §2 · p. 2
  • Bayesian and ensemble or multi-task approaches can relate to calibration, but the paper distinguishes them from direct post-processing calibration for semantic segmentation.The discussion notes that Bayesian uncertainty quantification may not achieve calibration, while ensembling and multi-task learning provide insight rather than directly addressing probability calibration.Source: §2 · p. 2

3. Methodology

The method calibrates a pre-trained segmentation network's confidence probabilities using a held-out validation set. It progresses from global to image-dependent to spatially local temperature scaling, and justifies temperature scaling through an entropy result for overconfident networks.

3.1. Problem Statement

  • The segmentation network produces logits and softmax confidence probabilities at each location; calibration aims to make confidence reflect the probability of a correct prediction.The paper extends classification calibration measures to assess semantic segmentation.Source: §3.1 · pp. 2–3

3.2. Calibration Setup

  • The calibration model is trained on a held-out validation set, using cross-entropy, to improve the pre-trained network's probability outputs on test data.Training, validation, and test sets are disjoint.Source: §3.2 · p. 3

3.3. TS for Probability Calibration

  • Standard TS uses one positive global temperature, while IBTS learns an image-dependent temperature with a CNN, addressing variation between images but not spatial differences within them.The paper's figures contrast local temperature values and reliability patterns across image regions.Source: §3.3 · p. 3

3.4. Local TS for Probability Calibration

  • Because confidence behavior can vary across locations, global and image-based temperatures cannot account for spatial variation; LTS uses a local temperature map.The paper illustrates localized differences in optimal temperature values and local reliability patterns.Source: §3.4 · p. 3, Figure 1 · p. 3, Figure 2 · p. 4
  • LTS assigns a positive, image- and location-dependent temperature, preserving class-probability ordering and therefore segmentation accuracy.Temperatures above 1 reduce confidence, while values below 1 increase it; the limits approach maximum or minimum uncertainty, respectively.Source: §3.4 · p. 4
  • The architecture scales the segmentation network's logit map using temperatures generated from the image and logits, then applies softmax to obtain calibrated probabilities.The temperature map is produced by an optimization or prediction unit.Source: §3.4 · p. 4, Figure 3 · p. 4

3.5. Theoretical Justification

  • The paper argues that NLL training can drive overconfidence because it continues pushing confidence upward even after segmentation error reaches zero.For calibration, the paper instead optimizes calibration parameters to recover probability distributions that describe observed errors.Source: §3.5 · pp. 4–5
  • For overconfident segmentation networks, minimizing negative log-likelihood with TS, IBTS, or LTS yields solutions that also maximize the entropy of calibrated probabilities under the stated condition.The paper presents this as a theoretical justification for its temperature-scaling variants.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate calibration on COCO, CamVid, and LPBA40, then test whether calibration helps downstream multi-atlas label fusion. LTS generally performs best across calibration regions, with slight segmentation improvements in label fusion.

4. Experiments

  • Evaluation uses five classification-derived calibration metrics, treating each pixel or voxel as a sample and measuring whole-image, boundary, and local-patch performance.Boundary regions extend up to two pixels or voxels from label edges; Local-Avg and Local-Max summarize average and worst-case performance across ten patches.Source: §4 · pp. 5–6

4.1. FCN semantic segmentation on COCO

  • On COCO, all methods except VS significantly improve calibration in Boundary regions, while LTS calibrates both whole-image and local regions better than alternatives.The reported segmentation mean IoU is 63.7%; local performance is assessed using average and worst-case patch calibration.Source: §4.1 · p. 6
  • MCE can remain large after calibration because a small number of annotation errors, especially at object boundaries, can produce near-total confidence–accuracy gaps.The authors therefore expect MCE values to be high relative to classification calibration results, even when LTS improves them.Source: §4.1 · p. 6

4. Experiments

  • LTS generally achieves the best performance on almost all metrics across All, Boundary, and Local regions in the four segmentation tasks.Table 1 reports calibration results; the COCO, CamVid, and LPBA40 analyses describe consistent advantages over other methods.Source: §4.1 · p. 6, Table 1 · p. 7, §4.2 · p. 7

4.2. Tiramisu semantic segmentation on CamVid

  • On CamVid, LTS calibrates All and local regions and generally outperforms TS and IBTS; adding it to MMCE or FL further improves calibration except in FL Boundary regions.CamVid contains consistent 11-class street scenes, and its four calibration metrics are lower than those reported for COCO.Source: §4.2 · p. 7

4.3. U-Net segmentation on LPBA40

  • On LPBA40 U-Net outputs, LTS performs best on most metrics and consistently leads in Local-region analysis, while TS and IBTS perform relatively similarly.The authors suggest affine registration to a common atlas space reduces image variation and makes calibration easier for all three methods.Source: §4.3 · pp. 7–8

4.4. Downstream MAS label fusion on LPBA40

  • For multi-atlas calibration, LTS outperforms IBTS, which outperforms TS, with statistically significant differences.The authors attribute the clearer separation partly to sufficient training data and reduced data variance after flexible registration.Source: §4.4 · p. 8
  • LTS calibration yields slight, consistent improvements in joint label fusion, although the results remain far from the two theoretical upper bounds.Table 2 reports that LTS changes more voxels than TS and IBTS and improves the balance of correct versus incorrect voxel conversions.Source: §4.4 · p. 8, Table 2 · p. 8

5. Conclusion and Future Work

The paper introduces local temperature scaling for spatially varying probability calibration in multi-label semantic segmentation. Across COCO, CamVid, and LPBA40, it reports improved calibration and slight benefits for downstream joint label fusion.

5. Conclusion and Future Work

  • LTS predicts spatially varying probability calibration for multi-label semantic segmentation and is reported to outperform approaches that cannot account for spatially varying miscalibration.The conclusion describes it as a general temperature scaling method.Source: §5 · p. 8
  • Experiments show LTS benefits standard and jointly trained segmentation models, while multi-atlas brain segmentation results show joint label fusion can improve with calibrated probabilities.The authors report improved segmentation results via joint label fusion.Source: §5 · p. 8
  • Future work could extend LTS to bin-wise calibration as in [30] or class-conditioned distributions as in [51].Source: §5 · p. 8

Supplementary Material

The supplementary material extends the paper’s related work, model description, and evaluation analysis. It reports that LTS generally performs best among the baselines and explains how calibration relates to downstream multi-atlas label fusion.

B. Networks for LTS and IBTS

  • The LTS and IBTS models use a tree-like convolutional network, modified from, with learned leaf filters and learned sigmoid-weighted mixtures at internal nodes.IBTS outputs one temperature per image, whereas LTS outputs a temperature map; the tree architecture is not claimed as a contribution.Source: Supplementary Material · p. 13, B. Networks for LTS and IBTS · pp. 14–15

Supplementary Material

  • The remaining appendices add local reliability examples, entropy analysis, metric definitions, evaluation-region illustrations, and results across local patch sizes.Source: Supplementary Material · p. 13
  • Additional evaluations in Appx. J are consistent with the paper’s conclusion that LTS generally works best among the baseline methods.Appx. I attributes differences in result magnitudes across datasets to their differing levels of variation, largest for COCO and smallest for LPBA40.Source: Supplementary Material · p. 13
  • Appx. K explains that VoteNet+ joint label fusion is sensitive to accurate probabilities and that improved calibration improves fused segmentation results.Source: Supplementary Material · p. 13

A. Additional Related Work

  • Appx. A contrasts the paper’s direct prediction of calibration parameters for pre-trained segmentation models with uncertainty approaches whose calibration can depend on assumptions.It discusses Bayesian models, ensembles, Monte Carlo dropout, and Gaussian Processes.Source: A. Additional Related Work · p. 14

C. Dataset Description and Implementation Details

The experiments cover COCO, CamVid, and LPBA40 using segmentation and calibration setups described for each task. Local reliability diagrams show LTS outperforming TS and IBTS, and the supplementary analysis connects temperature scaling to entropy.

C. Dataset Description and Implementation Details

D. Local Reliability Diagrams

  • LTS performs better than TS and IBTS in both whole-image and local-patch reliability diagrams for the CamVid Tiramisu experiment.The observation is consistent with the quantitative results in Table 1.Source: D. Local Reliability Diagrams · p. 17

E. Temperature Scaling from Entropy Point of View

  • The entropy analysis frames TS, IBTS, and LTS as solutions to entropy maximization or minimization under different constraints.It states that temperature scaling determined from a dataset yields an equilibrium between NLL and entropy that empirically corresponds to a well-calibrated state.Source: E. Temperature Scaling from Entropy… · p. 17

E.1. Minimize NLL with (Local) Temperature Scaling

This section analyzes NLL minimization under global, image-based, and local temperature scaling, then relates confidence states and entropy extremes to those solutions. It presents results for the optimal scaling parameters and the equal-probability entropy maximum.

E.1. Minimize NLL with (Local) Temperature Scaling

  • The theorem characterizes optimal TS, IBTS, and LTS temperature values for an NLL minimization problem over logit maps and label maps.For LTS, the temperature is specified at each position.Source: §E.1 · pp. 18–19

E.2. Overconfidence and Underconfidence

  • The section defines overconfidence, underconfidence, and balance at dataset, image, and position levels, and derives constraints used for TS, IBTS, and LTS.It describes overconfidence through NLL being at least entropy, and underconfidence through NLL generally being at most entropy.Source: §E.2 · pp. 20–21

E.3. Weighted Averaged Logits and Entropy Extremes

  • Equal probability across labels is the unique solution to the entropy-maximization problem.Source: §E.3 · p. 23

E.4. Entropy Extremes Under Constraints

The appendix derives entropy-optimal solutions for TS, IBTS, and LTS, whose constraints impose increasingly local assumptions. It also presents an entropy-minimization formulation for the same scaling approaches.

E.4. Entropy Extremes Under Constraints

  • Under constraints A, B, and C, TS, IBTS, and LTS are respectively the unique solutions to entropy maximization, with corresponding optimal temperature values.The appendix also formulates entropy minimization for the same three approaches.Source: §E.4 · pp. 24, 29, 33
  • Theorem 2 uses KKT conditions, while Theorem 3 uses gradients and an explicit softmax probability expression.The stated proof choice reflects Theorem 2's concave objective and self-contained constraints.Source: §E.4 · pp. 24, 35
  • Constraint A applies across all samples and the image space, B within each image, and C at each location; their restrictiveness is C > B > A and model complexity is LTS > IBTS > TS.Source: §E.4 · p. 29

E.5. (Local) Temperature Scaling Drives NLL and Entropy to an Equilibrium

For both overconfident and underconfident segmentation networks, post-hoc TS, IBTS, and LTS bring NLL and entropy into equilibrium. The equilibrium applies at different scopes for the three methods.

E.5. (Local) Temperature Scaling Drives NLL and Entropy to an Equilibrium

  • For overconfident networks, minimizing NLL also maximizes entropy; for underconfident networks, minimizing NLL also minimizes entropy.Theorem 4 states that the resulting solutions reach an NLL–entropy equilibrium in both cases.Source: §E.5 · p. 35
  • At the overconfidence optimum, NLL is minimized as entropy is maximized, and the two are equal, transferring the state to a balanced one.Source: §E.5 · p. 36
  • For underconfident networks, the optimum simultaneously minimizes NLL and entropy, with the two equal at equilibrium and the state becoming balanced.Source: §E.5 · p. 36
  • TS reaches equilibrium on the validation dataset, while IBTS and LTS reach it for each image and each location, respectively.Source: §E.5 · p. 36

F. Evaluation Metrics for Semantic Segmentation

The section defines calibration measures and segmentation-surface metrics, then specifies boundary-focused and background-excluding evaluation regions for brain segmentation.

F. Evaluation Metrics for Semantic Segmentation

G. Example of Boundary Region and All Region

  • The Boundary region includes label boundaries and voxels up to two voxels away; multi-atlas evaluation uses original MR labels because VoteNet+ boundaries are sparse and thin.The All region excludes background, and this regional evaluation matches the U-Net experiment.Source: G. Example of Boundary Region and All… · p. 38

H. Patch Size vs Metrics Results

Patch-size experiments show that smaller patches tend to worsen calibration, while LTS can outperform TS and IBTS by capturing spatial variation.

H. Patch Size vs Metrics Results

I. Dataset Variations

Image variation and quantitative results follow the same ordering across the experiments, from COCO with FCN as the weakest to LPBA40 with VoteNet+ as the best.

I. Dataset Variations

  • COCO with FCN has the weakest quantitative results, while LPBA40 with VoteNet+ has the best; CamVid with Tiramisu and LPBA40 with UNet fall between them.The passage says this ordering follows the datasets’ image-variation pattern.Source: §I · p. 39

J. Additional Quantitative Results

The additional results report stronger calibration with LTS than several calibration baselines. In multi-atlas segmentation, the paper uses VoteNet+ probabilities to examine whether calibration improves label fusion and segmentation accuracy.

K. Multi-atlas Segmentation and Joint Label Fusion

  • VoteNet+ images are registered to a target by deformable registration and show less variation than UNet images, which are affine pre-registered to a common atlas space.Source: Figure 8 · p. 40

J. Additional Quantitative Results

  • LTS works significantly better than TS, IsoReg, ETS, VS, and DirODIR; Table 3 reports its best performance on almost all metrics across All, Boundary, and Local regions.MMCE and MMCE+LTS results use downsampled images and cannot be directly compared with the other methods.Source: J. Additional Quantitative Results · p. 39, Table 3 · p. 41

K. Multi-atlas Segmentation and Joint Label Fusion

  • Joint label fusion down-weights pairs of atlases that consistently make similar errors, addressing correlated errors that spatially varying weighted voting does not consider.Source: K. Multi-atlas Segmentation and Joint… · p. 41
  • Multi-atlas segmentation depends heavily on estimating whether each atlas has the target label; VoteNet+ estimates these probabilities for calibration experiments on fusion and segmentation accuracy.The atlases are warped into target-image space before their candidate segmentations are fused.Source: K. Multi-atlas Segmentation and Joint… · pp. 39, 41, 42

Related papers

Loading 2008.05105v2…