Source-linked AI summary
A Critical Synthesis of Uncertainty Quantification and Foundation Models for Semantic Segmentation
Steven Landgraf, Joceline Hinz, Markus Ulrich
TL;DR
Uncertainty quantification remains underexplored for foundation models in semantic segmentation, despite their reliability challenges in safety-critical settings. This paper systematically benchmarks four UQ strategies with a SAM2–DPT model, finding trade-offs between accuracy, reliability, and computational cost rather than a single best method.
Problem
Systematic evidence on integrating existing uncertainty-quantification methods with foundation models such as SAM for semantic segmentation remains limited, despite reliability concerns in safety-critical applications.
Method
The study combines a pretrained SAM2 image encoder with a lightweight DPT decoder and benchmarks Monte Carlo Dropout, Deep Sub-Ensemble, Test-Time Augmentation, and Evidential Deep Learning across multiple domains.
Results
Competitive segmentation accuracy does not ensure reliable calibration, while UQ strategies exhibit distinct trade-offs in uncertainty quality, predictive performance, and inference speed.
Takeaways & Limitations
There is no one-size-fits-all UQ strategy, so future uncertainty-aware foundation models must jointly address predictive performance, reliability, and efficiency.
Abstract
from arXiv · showhide
Foundation models are increasingly breaking what seemed to be impossible not long ago by enabling unprecedented accuracy and cross-domain generalization. Yet their lack of interpretability, tendency to be overconfident, and sensitivity to real-world domain shifts pose critical challenges for safety- and mission-critical applications. Uncertainty quantification (UQ) offers a principled way to address these issues, but its integration into segmentation foundation models has yet to be explored. In this paper we present the first systematic evaluation of UQ methods applied to a foundation model for semantic segmentation. We fine-tune a lightweight DPT decoder on top of the pretrained SAM2 encoder to establish a simple yet competitive baseline and benchmark four representative UQ approaches - Monte Carlo Dropout, Deep Sub-Ensemble, Test-Time Augmentation, and Evidential Deep Learning - across Cityscapes, NYUv2, and two challenging out-of-domain settings. Our analysis compares segmentation accuracy, calibration, uncertainty quality, and inference time, revealing clear trade-offs between predictive performance, reliability, and computational cost. These results highlight both the promise and the current limitations of uncertainty-aware foundation models, pointing to the need for future work that jointly optimizes accuracy, robustness, and efficiency for real-world deployment.
1. Introduction
Foundation models promise unprecedented accuracy, generalization, and zero-shot segmentation across diverse domains, but their limited interpretability, overconfidence, out-of-domain failures, and domain sensitivity create safety challenges. This study addresses these limitations by systematically evaluating uncertainty quantification methods for segmentation foundation models and comparing accuracy, calibration, uncertainty quality, and inference time.
- Motivation: Foundation models extend semantic segmentation from CNNs and vision transformers to internet-scale pretrained systems promising unprecedented accuracy, generalization, and zero-shot capabilities.Semantic segmentation assigns class labels to every image pixel, while foundation models mark a recent evolution of the task.
- Challenges: Foundation models can lack interpretability, fail to recognize out-of-domain inputs, produce systematically overconfident outputs, and remain sensitive to adversarial perturbations.These limitations are particularly harmful in safety-critical applications.
- Uncertainty quantification: Reliable uncertainty estimates can mitigate overconfidence and domain-shift sensitivity, enhance interpretability, and flag unreliable predictions for human intervention or additional verification.The paper motivates UQ for applications including autonomous driving and medical imaging, where unreliable predictions can increase catastrophic-failure risk.
- Contributions: The study presents the first systematic evaluation of UQ methods for semantic-segmentation foundation models, covering Monte Carlo Dropout, Deep Sub-Ensemble, Test-Time Augmentation, and Evidential Deep Learning.The evaluation targets how different UQ methods interact with powerful foundation models.
- Contributions: A simple yet competitive baseline fine-tunes a DPT head on a pretrained SAM2 encoder and benchmarks in-domain and out-of-domain performance across multiple datasets.The broader comparison covers segmentation performance, calibration, uncertainty quality, and inference time, exposing trade-offs between accuracy, reliability, and computational cost.
2. Related Work
Related work traces semantic segmentation from CNNs through vision transformers and foundation models, alongside established deep-learning uncertainty-quantification methods. Although early studies have explored uncertainty-aware SAM applications, systematic evaluation of existing UQ methods with SAM remains an open gap.
- Semantic Segmentation: CNN-based segmentation progressed from fully convolutional pixel-wise classification to richer contextual and multi-scale architectures such as DeepLab and U-Net.DeepLab introduced dilated convolutions and atrous spatial pyramid pooling, while U-Net used an encoder-decoder design.
- Semantic Segmentation: Vision transformers brought transformer architectures into vision, with SegFormer combining a hierarchical transformer encoder and lightweight multilayer perceptron decoder for efficient segmentation.The passage identifies SegFormer as achieving impressive results and high efficiency.
- Uncertainty Quantification: Sampling-based uncertainty methods remain popular because of their ease of use and effectiveness, including Monte Carlo Dropout as a Bayesian approximation using dropout.The overview situates Monte Carlo Dropout among a broader range of predictive-uncertainty techniques.
- Uncertainty-aware Foundation Models: Uncertainty-aware foundation-model research is emerging, with prior work spanning monocular depth estimation, medical-imaging applications of SAM, USAM, and SUM.Landgraf et al. identify semantic segmentation as an open extension opportunity, while USAM estimates expected uncertainty and SUM quantifies uncertainty in SAM-generated pseudo-labels.
- Research Gap: Despite tailor-made advances, no systematic study has evaluated existing UQ methods in combination with SAM, a de-facto foundation model for large-scale segmentation.This gap motivates the paper’s systematic investigation of uncertainty quantification for semantic-segmentation foundation models.
3. Methodology
The methodology combines a pretrained SAM2 image encoder with a lightweight DPT decoder for semantic segmentation and integrates four representative uncertainty-quantification techniques. The methods span stochastic sampling, decoder ensembles, evidential modeling, and inference-time augmentation while preserving a simple architecture and training objective.
- Baseline architecture: The baseline uses a hybrid architecture combining the SAM2 image encoder with a DPT decoder for semantic segmentation.The design leverages SAM2’s pretrained hierarchical Vision Transformer Hiera features while keeping the remaining architecture simple.
- UQ methods: The study evaluates four representative UQ techniques: Monte Carlo Dropout, Deep Sub-Ensemble, Evidential Deep Learning, and Test-Time Augmentation.These methods represent Bayesian, ensemble, deterministic, and test-time augmentation approaches.
- Monte Carlo Dropout: Monte Carlo Dropout estimates predictive uncertainty by keeping dropout active during inference and averaging the resulting stochastic softmax predictions.Uncertainty is quantified from predictive variation, including standard deviation, variance, or predictive entropy.
- Deep Sub-Ensemble: Deep Sub-Ensemble shares the SAM2 encoder across independently initialized DPT decoders, cycling optimization across decoder heads to increase diversity efficiently.At inference, predictions and uncertainty are computed across the decoder heads.
- Evidential Deep Learning: Evidential Deep Learning predicts nonnegative class evidence that parameterizes a Dirichlet distribution, using evidential and KL-regularized losses to calibrate uncertainty.A two-head architecture separates segmentation logits from evidential uncertainty parameters and improved segmentation performance over a single shared head.
- Test-Time Augmentation: Test-Time Augmentation estimates uncertainty only during inference by averaging predictions from original and augmented images, with uncertainty derived from predictive variation or entropy.TTA does not modify model parameters or the training protocol.
4. Experimental Setup
The experiments use a unified training and evaluation setup spanning indoor/outdoor and in-domain/out-of-domain segmentation datasets. They compare UQ methods under shared augmentations, sampling configurations, and accuracy, calibration, and uncertainty-quality metrics.
- Training configuration: Models train for 100 epochs with AdamW, a 3 × 10−5 base learning rate, polynomial decay, a 10× decoder learning-rate multiplier, and cross-entropy loss.The decoder is newly initialized while the SAM2 encoder is pretrained, motivating the decoder learning-rate multiplier.
- Datasets: Experiments evaluate Cityscapes and NYUv2, plus challenging Rainy-Cityscapes and Foggy-Cityscapes variants for out-of-domain testing.The datasets jointly cover outdoor and indoor conditions as well as in-domain and out-of-domain settings.
- Data augmentation: Training augmentations use random scaling from 0.5 to 2.0, dataset-specific cropping, and random horizontal flipping with a 50 % probability.Crop sizes are 768×768 px for Cityscapes and 480×640 px for NYUv2.
- UQ configurations: TTA uses vertical and horizontal flips plus scaling at inference, while MCD samples ten times with 20% dropout and DSE uses ten DPT heads.Vertical flipping may overestimate uncertainty by altering spatial priors, but it was retained because it performed best.
- Evaluation metrics: Evaluation reports mIoU, ECE, p(acc.|cer.), and p(unc.|ina.), with high values expected for the two uncertainty-quality metrics.The uncertainty threshold is the median uncertainty of each image, and predictive entropy is used instead of predictive variance.
5. Experiments
Experiments show that a lightweight SAM2-DPT baseline is competitive for segmentation, but its calibration and uncertainty quality vary across datasets and domains. UQ methods introduce trade-offs among accuracy, reliability, and inference cost, with no single best strategy.
- Comparison with SOTA: The lightweight SAM2-DPT baseline achieves performance comparable to recent state-of-the-art methods on Cityscapes and NYUv2 without architectural changes or complex training.Subsequent experiments use the SAM2-Tiny backbone.
- In-Domain Evaluation: On Cityscapes, the baseline offers competitive mIoU, strong calibration, and fastest inference, whereas DSE improves calibration and uncertainty quality at lower mIoU and substantially higher cost.The baseline has the second-worst uncertainty quality, with EDL performing worst across all metrics.
- In-Domain Evaluation: On NYUv2, the baseline has highest segmentation performance and best uncertainty quality but poorest calibration; TTA has best calibration, while EDL performs poorly beyond its second-fastest inference.These results warrant caution because overall segmentation scores are low.
- Out-of-Domain Evaluation: Segmentation performance drops markedly on Rainy- and Foggy-Cityscapes, demonstrating vulnerability to domain shifts.The decline occurs relative to the in-domain experiments.
- Out-of-Domain Evaluation: On Rainy-Cityscapes, TTA leads mIoU, calibration, and improvement in uncertainty quality, while on Foggy-Cityscapes it leads mIoU but DSE provides the best uncertainty scores.MCD trades slightly higher Rainy-Cityscapes mIoU for substantially worse calibration, and EDL performs worst across out-of-domain metrics.
- Key Findings: Overall, predictive accuracy does not guarantee reliability, and UQ strategies trade uncertainty quality and calibration against inference cost without producing a universally best method.DSE provides high uncertainty quality at significant speed cost; TTA often improves on the baseline but is computationally expensive, while MCD and EDL are generally less promising.
6. Conclusion
The study systematically evaluates multiple UQ methods for a SAM2-based semantic-segmentation foundation model, showing that lightweight decoding can achieve competitive accuracy while reliability and efficiency vary markedly across strategies. The findings indicate that no single UQ approach is universally optimal.
- The paper presents the first systematic evaluation of multiple UQ methods for a SAM2-based foundation model for semantic segmentation.
- Competitive segmentation accuracy is attainable with a lightweight decoder across Cityscapes, NYUv2, and two out-of-domain settings.
- Reliability and efficiency differ markedly between UQ strategies, indicating that no one-size-fits-all solution exists yet.