Source-linked AI summary
Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
Jishnu Mukhoti, Yarin Gal
TL;DR
Bayesian semantic segmentation needs evaluation metrics that assess uncertainty estimates because ground-truth uncertainties are unavailable and conventional calibration metrics miss epistemic uncertainty. The paper proposes three segmentation-specific metrics, builds Bayesian DeepLab-v3+ variants with MC and Concrete dropout, and evaluates them on Cityscapes. Concrete dropout outperforms MC dropout on the reported segmentation metrics, while the proposed metrics capture random-guessing behavior that ECE and MCE do not.
Problem
Ground-truth uncertainties are unavailable for Bayesian semantic segmentation, while conventional calibration metrics cannot capture epistemic uncertainty or reliably detect random guessing.
Method
The paper proposes three semantic-segmentation-specific metrics and evaluates probabilistic DeepLab-v3+ variants using MC dropout and Concrete dropout on Cityscapes.
Results
Concrete dropout performs better than MC dropout on pixel accuracy, mean accuracy, and mean IOU; the proposed metrics capture random guessing that ECE and MCE miss.
Takeaways & Limitations
The study provides benchmarks for comparing Bayesian semantic segmentation models and their uncertainty quantification.
Takeaways & Limitations
The proposed metrics do not yet measure how segmentation outputs and uncertainty estimates affect safe, correct autonomous-driving decisions or other downstream applications.
Abstract
from arXiv · showhide
Deep learning has been revolutionary for computer vision and semantic segmentation in particular, with Bayesian Deep Learning (BDL) used to obtain uncertainty maps from deep models when predicting semantic classes. This information is critical when using semantic segmentation for autonomous driving for example. Standard semantic segmentation systems have well-established evaluation metrics. However, with BDL's rising popularity in computer vision we require new metrics to evaluate whether a BDL method produces better uncertainty estimates than another method. In this work we propose three such metrics to evaluate BDL models designed specifically for the task of semantic segmentation. We modify DeepLab-v3+, one of the state-of-the-art deep neural networks, and create its Bayesian counterpart using MC dropout and Concrete dropout as inference techniques. We then compare and test these two inference techniques on the well-known Cityscapes dataset using our suggested metrics. Our results provide new benchmarks for researchers to compare and evaluate their improved uncertainty quantification in pursuit of safer semantic segmentation.
1. Introduction
The paper addresses the lack of task-specific ways to evaluate uncertainty in Bayesian semantic segmentation. It proposes three metrics, implements probabilistic DeepLab-v3+ variants, and benchmarks MC dropout against Concrete dropout on Cityscapes.
- Motivation: Bayesian semantic segmentation produces pixel-wise predictions and uncertainty estimates, which are important for applications such as autonomous driving.Conventional deep learning models generally provide point estimates without confidence information.
- Contribution: The paper proposes three specialized metrics for evaluating Bayesian models designed for semantic segmentation.These metrics are intended to compare and benchmark uncertainty estimates rather than only segmentation accuracy.
- Motivation: Because ground-truth uncertainties are unavailable, evaluation must relate uncertainty estimates to prediction accuracy and desired confidence behavior.The paper specifies that confident predictions should be accurate, while uncertain predictions may or may not be accurate.
- Method: The authors create probabilistic DeepLab-v3+ models using MC dropout and Concrete dropout, both of which provide uncertainty estimates alongside segmentation results.DeepLab-v3+ is selected as a state-of-the-art semantic segmentation architecture.
- Method: The evaluation system passes inputs through a Bayesian neural network, then combines ground-truth labels, predictions, and uncertainties to compute the proposed metrics.The system overview describes the evaluation pipeline at a high level.
2. Related Work
Prior work developed semantic segmentation architectures and approximate Bayesian inference methods, but existing uncertainty evaluations were insufficient for quantitatively comparing Bayesian segmentation models.
- Semantic segmentation: Semantic segmentation research includes encoder-decoder architectures and dilated convolutions; DeepLab combines dilated convolutions with a decoder.The paper adopts DeepLab-v3+ as its base network.
- Approximate inference: Bayesian modelling represents uncertainty over neural-network parameters, but exact posterior inference is computationally intractable for large networks.Approximate approaches include MCMC and variational inference.
- Approximate inference: The paper uses MC dropout and Concrete dropout as variational approximate-inference techniques for Bayesian neural networks.Both methods are dropout-based and require limited architectural changes.
- Existing uncertainty metrics: Calibration metrics such as ECE and MCE use softmax probabilities and cannot capture epistemic uncertainty.Temperature scaling can make deterministic and probabilistic models appear equally calibrated, limiting these metrics for detecting random guessing.
- Existing uncertainty metrics: Bayesian SegNet demonstrated qualitative accuracy-versus-uncertainty checks, but those checks did not quantitatively compare Bayesian segmentation models.The paper positions its proposed measures as filling this comparison gap.
3. Bayesian DeepLab
The paper adapts DeepLab-v3+ into Bayesian semantic-segmentation models using dropout-based approximate inference, producing predictions and pixel-wise uncertainty estimates. It evaluates predictive entropy and mutual information as uncertainty measures.
- Bayesian DeepLab: DeepLab-v3+ is implemented with Xception as the base network, using atrous convolutions and a decoder for fine-grained segmentation.The architecture restores original image dimensions through its decoder.
- Approximate inference: MC dropout keeps trained-network dropout active during testing, treating stochastic forward passes as posterior samples.Multiple passes yield a mean prediction and use variance as an uncertainty estimate.
- Approximate inference: Concrete dropout provides an alternative dropout-based inference method that tunes dropout rates during training.Both methods require minimal architectural changes and are compared within Bayesian DeepLab.
- Uncertainty metrics: The Bayesian variants generate pixel-wise uncertainty maps alongside semantic segmentation predictions.The evaluation system uses these maps to assess uncertainty at the pixel level.
- Uncertainty metrics: Predictive entropy measures combined epistemic and aleatoric uncertainty, while mutual information captures epistemic or model uncertainty.Both metrics are approximated using Monte Carlo samples and class probabilities.
4. Performance Evaluation Metrics
The paper evaluates Bayesian segmentation models by comparing prediction accuracy with uncertainty over local image patches. It defines conditional metrics and combines favorable patch outcomes into PAvPU.
- Metric assumptions: The metrics assume confident predictions should be accurate and inaccurate predictions should be uncertain, while noting the converse need not hold.High epistemic uncertainty can occur for infrequent classes even when predictions remain accurate.
- Patch construction: Patch accuracy is computed from predicted and actual labels, while average patch uncertainty is computed from the corresponding uncertainty-map region.Patches use dimensions w > 1 to capture spatially coherent uncertain regions rather than isolated pixels.
- Conditional metrics: The evaluation forms a confusion matrix over accurate/certain, accurate/uncertain, inaccurate/certain, and inaccurate/uncertain patches.Conditional probabilities report p(accurate|certain) and p(uncertain|inaccurate).
- PAvPU: PAvPU combines accurate-certain and inaccurate-uncertain patches as PAvPU = (nac + niu) / (nac + nau + nic + niu).Higher values of the proposed metrics indicate better performance.
- Parameter sensitivity: Metric values depend on the accuracy threshold, uncertainty threshold, and patch dimensions, which the paper varies experimentally.The authors use Figures 3 and 4 to examine these parameter effects.
5. Experiments and Results
The experiments evaluate Bayesian DeepLab variants on Cityscapes using segmentation metrics and proposed uncertainty metrics. Concrete dropout generally outperforms MC dropout, while dropout-based models outperform the deterministic baseline on uncertainty evaluation.
- Experimental setup: The evaluation compares MC dropout and Concrete dropout Bayesian models with a deterministic DeepLab-v3+ baseline on Cityscapes.Segmentation is assessed with pixel accuracy, mean accuracy, and mean IOU; uncertainty is evaluated using the three proposed metrics.
- Semantic segmentation performance: Concrete dropout outperforms MC dropout on pixel accuracy, mean accuracy, and mean IOU.Its mean IOU is 79.12, compared with 78.05 for the other Bayesian variant; DeepLab-v3+ reaches 79.14 under the stated comparison setup.
- Uncertainty evaluation: The proposed metrics vary with accuracy threshold, uncertainty threshold, and patch dimensions.The experiments use 4 × 4 patches, a 0.5 accuracy threshold, and the validation-set mean uncertainty as the uncertainty threshold.
- Uncertainty evaluation: Dropout-based models outperform the deterministic model on all three proposed uncertainty metrics, with Concrete dropout consistently best.The proposed metrics are p(accurate|certain), p(uncertain|inaccurate), and PAvPU.
- Threshold analysis: At a 0% uncertainty threshold, p(accurate|certain) is undefined and p(uncertain|inaccurate) equals 1; at 100%, the former metrics reduce to patch accuracy and the latter equals 0.The PAvPU values at 0% and 100% sum to 1, with the 100% value significantly larger.
- Comparison with calibration metrics: After temperature scaling, ECE and MCE do not distinguish the models, whereas the proposed metrics capture differences in uncertainty behavior.The paper therefore recommends using the proposed uncertainty metrics alongside conventional accuracy measures such as mean IOU.
6. Conclusions
The work develops metrics for evaluating Bayesian semantic-segmentation models and evaluates probabilistic DeepLab-v3+ variants on Cityscapes, establishing benchmarks for future comparisons.
- The authors develop metrics specifically for evaluating Bayesian models on semantic segmentation.
- Two probabilistic DeepLab-v3+ variants are evaluated using the proposed metrics on Cityscapes.The variants use MC dropout and Concrete dropout inference techniques, as described earlier in the paper.
- The benchmarks are intended to support future comparisons of Bayesian semantic-segmentation models.
- Future work would evaluate Bayesian models by how effectively segmentation outputs and uncertainty estimates support safe, correct autonomous-driving decisions.The authors also suggest extending this direction to other downstream applications where semantic segmentation is an intermediate tool.
A. Bayesian DeepLab Network Architecture
The Bayesian DeepLab architecture uses an Xception-based, fully convolutional network organized into entry, middle, and exit flows, with ASPP and a decoder supporting multiscale recognition and output resizing.
- The Bayesian DeepLab network uses Xception as its base network and passes inputs through entry, middle, and exit flows.
- The network replaces pooling layers with stride-2 separable convolutions to support dense predictions.
- The fully convolutional design allows segmentation of images with arbitrary sizes.
- The middle flow contains 16 modules rather than the 8 modules described in the original Xception architecture.
- The ASPP module recognizes objects at different scales, while the Decoder resizes outputs to the original input dimensions.
B. Training Infrastructure
The networks are trained on Cityscapes with specified architectural and optimization settings, using 90,000 iterations on eight NVIDIA Tesla P100-SXM2 GPUs over approximately three days.
- Cityscapes provides 5,000 urban-scene images from 50 cities, split into 2,975 training, 500 validation, and 1,525 test images.The images have dimensions 2048 × 1024.
- Training uses ASPP atrous rates [6] [12], output stride 16, decoder output stride 4, 512 × 512 crops, and batch size 16.
- Concrete dropout uses a weight regulariser of 1e −8 and a dropout regulariser of 1/(n × h × w).Here, n is the number of training images and h and w are the image height and width.
- All networks are trained for 90,000 iterations on 8 NVIDIA Tesla P100-SXM2 GPUs and finish in approximately 3 days.