Source-linked AI summary
Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding
Alex Kendall, Vijay Badrinarayanan, Roberto Cipolla
TL;DR
Semantic segmentation needs reliable pixel-wise predictions and uncertainty estimates for scene understanding and decision making. Bayesian SegNet uses test-time dropout and Monte Carlo sampling to produce probabilistic segmentations, improving accuracy across several architectures and showing particular effectiveness on smaller datasets.
Problem
Deep-learning segmentation methods lacked probabilistic pixel-wise outputs with model-uncertainty measures, although such confidence information matters for scene-understanding decisions.
Method
Bayesian SegNet extends convolutional encoder-decoder networks by sampling test-time dropout models to approximate posterior distributions of pixel class probabilities without additional parameterization.
Results
2-3% improvement in segmentation accuracy is achieved across SegNet, FCN, and Dilation Network, with larger gains on smaller datasets and top benchmark results on CamVid and SUN RGB-D.
Takeaways & Limitations
Model uncertainty provides a practical confidence measure for segmentation outputs and is especially effective when training data are limited.
Takeaways & Limitations
Using depth would require architectural modifications and careful post-processing to fill missing measurements, which is outside the paper’s scope.
Abstract
from arXiv · showhide
We present a deep learning framework for probabilistic pixel-wise semantic segmentation, which we term Bayesian SegNet. Semantic segmentation is an important tool for visual scene understanding and a meaningful measure of uncertainty is essential for decision making. Our contribution is a practical system which is able to predict pixel-wise class labels with a measure of model uncertainty. We achieve this by Monte Carlo sampling with dropout at test time to generate a posterior distribution of pixel class labels. In addition, we show that modelling uncertainty improves segmentation performance by 2-3% across a number of state of the art architectures such as SegNet, FCN and Dilation Network, with no additional parametrisation. We also observe a significant improvement in performance for smaller datasets where modelling uncertainty is more effective. We benchmark Bayesian SegNet on the indoor SUN Scene Understanding and outdoor CamVid driving scenes datasets.
1. Introduction
Semantic segmentation supports scene understanding and decision making, but existing deep-learning methods lack probabilistic outputs that quantify model uncertainty. Bayesian SegNet addresses this gap and reports improved performance, especially on smaller datasets.
- Motivation: Semantic segmentation labels images at the pixel level and supports applications including scene geometry, robotic interaction, and autonomous driving.Scenes vary substantially in pose and appearance, making pixel-wise understanding difficult.
- Motivation: Existing deep-learning segmentation methods did not produce probabilistic segmentations with a measure of model uncertainty.Confidence information is relevant when segmentation affects behavioral decisions, such as distinguishing pedestrians, cyclists, and street signs.
- Contribution: Bayesian SegNet extends deep convolutional encoder-decoder architectures into Bayesian networks that produce probabilistic pixel-wise segmentation without additional parameterization.Dropout at test time approximates the posterior by sampling Bernoulli-distributed network weights.
- Results: Bayesian SegNet sets the best-performing benchmark on CamVid Road Scenes and SUN RGB-D Indoor Scene Understanding datasets.The paper reports a larger improvement on smaller datasets such as CamVid.
- Results: 2-3% improvement in segmentation accuracy is achieved when uncertainty modelling is applied to SegNet, FCN, and Dilation Network.The comparison spans multiple state-of-the-art segmentation architectures.
- Results: Model uncertainty is evaluated as a measure of confidence in image segmentations and through analysis of factors associated with uncertain predictions.The paper frames uncertainty as useful for understanding when segmentation outputs can be trusted.
2. Related Work
Related work progressed from hand-crafted visual features to deep segmentation architectures and post-processing methods, but these systems generally lacked probabilistic uncertainty estimates. Bayesian inference and test-time dropout provide the conceptual basis for adding such estimates without extra parameters.
- Deep segmentation: Deep segmentation architectures such as SegNet, FCN, and Dilation Network became prominent core engines for pixel-wise labelling.SegNet supported end-to-end one-step training because of its lower parameterization.
- Post-processing: Post-processing methods use region proposals, conditional random fields, or recurrent neural networks to smooth outputs and improve label consistency.These methods augment rather than replace the core segmentation engine.
- Research gap: Existing segmentation methods did not generate probabilistic outputs with a measure of model uncertainty.This limitation motivates the Bayesian SegNet formulation.
- Bayesian neural networks: Bayesian neural networks model uncertainty by inferring distributions over network weights, but inference is difficult and often relies on posterior approximations.They can also be computationally expensive and increase parameterization without substantially increasing model capacity.
- Dropout: Dropout regularizes convolutional networks by randomly removing units during training, thereby sampling thinned networks.This is commonly used to reduce overfitting and feature co-adaptation.
- Dropout: Test-time dropout can impose a Bernoulli distribution over convolutional filter weights and provide Monte Carlo samples from an approximate model posterior without additional parameters.The paper applies this approach to pixel-wise semantic segmentation.
- Uncertainty estimation: Monte Carlo sampling yields a model-uncertainty distribution distinct from softmax scores, which represent relative class probabilities rather than overall uncertainty.The paper uses this distinction to motivate uncertainty estimation beyond standard classification outputs.
3. SegNet Architecture
Bayesian SegNet modifies the SegNet encoder-decoder architecture to produce probabilistic pixel-wise predictions. Its pipeline uses VGG-16-based encoders, reversed decoders, and test-time dropout sampling to estimate uncertainty.
- SegNet architecture: SegNet is a deep convolutional encoder-decoder architecture with nonlinear encoder layers, corresponding decoders, and a pixel-wise classifier.Encoders typically combine convolution, batch normalization, ReLU, and max-pooling operations.
- Bayesian extension: The Bayesian SegNet pipeline is trained end-to-end with stochastic gradient descent and obtains probabilistic outputs from test-time dropout samples.The architecture diagram shows the complete training and inference pipeline.
- Decoder: Max-pooling indices guide decoder upsampling, retaining class-boundary details while reducing the total number of model parameters.The model is trained end-to-end using stochastic gradient descent.
- Model variants: SegNet uses 13 VGG-16 convolutional encoder layers followed by 13 corresponding decoders, while SegNet-Basic uses four encoder and decoder layers with 64 features.SegNet-Basic provides a smaller model for analysis of the larger architecture.
4. Bayesian SegNet
Bayesian SegNet uses dropout-based approximate Bayesian inference to produce probabilistic segmentation and model uncertainty. Experiments examine posterior sampling, dropout placement, and performance against weight averaging on CamVid.
- Bayesian inference: Bayesian SegNet samples the posterior distribution over network weights with test-time dropout to obtain probabilistic softmax class predictions.The method uses variational inference with Bernoulli-distributed weights and fixed dropout probability pi = 0.5.
- Uncertainty estimation: The model uses posterior sample means for segmentation predictions and per-class variance, averaged across classes, for overall model uncertainty.This separates the predicted class probabilities from the uncertainty estimate.
- Probabilistic variants: Dropout placement is varied across encoder and decoder units because applying it throughout the network is too strong a regulariser and lowers test performance.The benchmark uses the Central Encoder-Decoder configuration, with dropout in the deepest half of the encoder and decoder units.
- Probabilistic variants: Deeper encoder-decoder layers are preferred for Bayesian weights, while lower layers remain deterministic because they extract more consistent basic features such as edges and corners.The paper associates deeper layers with higher-level shape and contextual relationships.
- Sampling versus weight averaging: After approximately 6 samples, Monte Carlo dropout outperforms weight averaging, and performance shows no significant improvement beyond approximately 40 samples.The comparison uses global segmentation accuracy on CamVid for SegNet and SegNet-Basic; sampling increases inference time.
5. Experiments
Experiments evaluate Bayesian SegNet across scene-understanding, indoor, and object-segmentation benchmarks, showing strong performance, broad architectural applicability, and informative uncertainty estimates. The method also incurs measurable inference-time overhead when Monte Carlo sampling is used.
- CamVid: Bayesian SegNet achieves the highest overall class-average and mean intersection-over-union scores on CamVid, setting new benchmarks on 7 of 11 classes.
- NYUv2: Bayesian SegNet is the top-performing RGB method on the NYUv2 subset of SUN RGB-D.
- SUN RGB-D: On SUN RGB-D, Bayesian SegNet outperforms previous benchmarks, including methods using depth, while operating with RGB-based predictions.The dataset contains 37 indoor scene classes and 5050 test images.
- Pascal VOC: Training Bayesian SegNet end-to-end without multi-stage training, extra datasets, object proposals, or CRF post-processing provides a more controlled Pascal VOC benchmark.
- General Applicability: Monte Carlo dropout improves segmentation performance by 2-3% across SegNet, FCN, and Dilation Network, demonstrating broad applicability.
- Uncertainty Analysis: Model uncertainty is higher at class boundaries and for visually ambiguous, rare, or challenging classes, while high-confidence predictions show very high accuracy.The reported uncertainty-confidence relationship supports using uncertainty as a measure of prediction confidence.
- Real Time Performance: Bayesian SegNet with 10 Monte Carlo samples runs at 90ms per frame on a Titan X GPU, compared with 35ms per frame for SegNet.Inference time depends on the implementation.
6. Conclusions
The paper presents Bayesian SegNet as a probabilistic semantic-segmentation framework that provides class-wise model uncertainty. Across benchmarks, it performs strongly, with uncertainty especially informative for boundaries, difficult objects, and smaller datasets.
- Bayesian SegNet produces probabilistic pixel-wise segmentation with a measure of model uncertainty for each class.
- The model is uncertain at object boundaries and for difficult or visually ambiguous objects.
- Bayesian SegNet provides a reliable uncertainty measure and is particularly effective when modelling smaller datasets.
- The model achieves the highest-performing results on CamVid road scenes and SUN RGB-D indoor scene understanding datasets.