Source-linked AI summary
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling
Vijay Badrinarayanan, Ankur Handa, Roberto Cipolla
TL;DR
Existing classification-oriented networks lack a learned route from low-resolution features to pixel dimensions, limiting semantic pixel-wise labelling. SegNet introduces a fully trainable encoder-decoder architecture that learns this mapping and uses increasing depth for broader context. The paper reports competitive outdoor and indoor scene accuracy without additional depth, video, or CRF post-processing.
Problem
Classification-oriented deep networks lack a mechanism to map low-resolution deepest feature maps back to input dimensions for pixel-wise labelling.
Method
SegNet uses a stack of encoders and corresponding decoders to learn a feed-forward mapping from input images through low-resolution features to pixel-wise semantic labels.
Results
SegNet achieves competitive numerical and qualitative accuracy on outdoor and indoor scenes without additional depth, video, or CRF cues.
Takeaways & Limitations
SegNet produces smooth segment labels by combining deep feature encoding, larger spatial context, and learned mapping from encoder features to semantic labels.
Takeaways & Limitations
Using pre-trained SegNet on KITTI samples without additional training resulted in poor per-pixel performance.
Abstract
from arXiv · showhide
We propose a novel deep architecture, SegNet, for semantic pixel wise image labelling. SegNet has several attractive properties; (i) it only requires forward evaluation of a fully learnt function to obtain smooth label predictions, (ii) with increasing depth, a larger context is considered for pixel labelling which improves accuracy, and (iii) it is easy to visualise the effect of feature activation(s) in the pixel label space at any depth. SegNet is composed of a stack of encoders followed by a corresponding decoder stack which feeds into a soft-max classification layer. The decoders help map low resolution feature maps at the output of the encoder stack to full input image size feature maps. This addresses an important drawback of recent deep learning approaches which have adopted networks designed for object categorization for pixel wise labelling. These methods lack a mechanism to map deep layer feature maps to input dimensions. They resort to ad hoc methods to upsample features, e.g. by replication. This results in noisy predictions and also restricts the number of pooling layers in order to avoid too much upsampling and thus reduces spatial context. SegNet overcomes these problems by learning to map encoder outputs to image pixel labels. We test the performance of SegNet on outdoor RGB scenes from CamVid, KITTI and indoor scenes from the NYU dataset. Our results show that SegNet achieves state-of-the-art performance even without use of additional cues such as depth, video frames or post-processing with CRF models.
1. Introduction
Semantic segmentation supports scene understanding, but adapting classification networks to pixel-wise labelling leaves low-resolution features without a learned path back to image dimensions. SegNet addresses this with a modular encoder-decoder stack that expands spatial context and produces smooth labels.
- Semantic segmentation supports scene geometry estimation, object support-relationship inference, and autonomous vehicle driving.
- Classification-oriented deep networks produce low-resolution deepest feature maps and lack a mechanism to map them back to input dimensions.
- SegNet learns a fully supervised encoder-decoder stack for pixel-wise labelling, inspired by probabilistic auto-encoders and feature-hierarchy learning.
- Adding deeper encoder-decoder pairs increases spatial context; a four-layer network with 7 × 7 kernels and 2 × 2 pooling reaches 106 × 106 pixels.
- SegNet predictions become smoother with additional layers and achieve accuracy comparable to or exceeding methods using CRFs.
- Experiments use outdoor RGB road-scene datasets and indoor RGBD scenes to evaluate quantitative performance.
2. Literature Review
Earlier segmentation methods commonly classified pixels using hand-engineered features and then smoothed predictions with CRFs. Deep approaches began replacing these pipelines, but classification networks still required ad hoc upsampling of low-resolution features, motivating learned decoding.
- Traditional methods classify center pixels or patches with Random Forests or Boosting using appearance, SfM, or combined cues.
- CRFs smooth noisy per-pixel classifier outputs through pair-wise or higher-order relationships.
- Patch-level label prediction improves Random Forest unaries, but thin structured classes remain poorly classified.
- Deep segmentation attempts adapted categorization networks by replicating deepest features or merging low-resolution predictions, producing blocky or otherwise limited outputs.
- SegNet draws on encoder-decoder networks that store pooling indices and use them to upsample feature maps during decoding.
- Learned upsampling from low-resolution feature maps is identified as the central topic of this paper.
3. SegNet Architecture and Learning Scheme
SegNet uses paired encoders and decoders to transform multi-channel inputs into pixel-wise class maps, while modular training progressively adds deeper pairs. Its memorized pooling indices preserve spatial structure during learned upsampling, and feature ablations show deeper representations are more category-tuned.
- SegNet Architecture and Learning Scheme: Each encoder applies convolution, ReLU, 2 × 2 max pooling, and down-sampling; each decoder upsamples with memorized pooling indices and trainable filters.
- SegNet Architecture and Learning Scheme: The final soft-max independently classifies every pixel and outputs a K-channel image, where K is the number of classes.
- SegNet Architecture and Learning Scheme: SegNet keeps feature counts constant at 64 per layer, avoiding parameter explosion while making deeper-pair convolutions faster at lower resolutions.
- SegNet Architecture and Learning Scheme: The architecture accepts arbitrary multi-channel images or feature maps, including RGB, RGBD, normals, and depth.
- SegNet Architecture and Learning Scheme: Modular training optimizes the first encoder-decoder pair, then inserts deeper pairs while holding preceding weights fixed; encoder and decoder weights are untied.
- 3.2. Visualizing the SegNet: Feature ablation maps selected activations back into pixel-label space by zeroing the remaining feature maps and decoding the result.
- 3.2. Visualizing the SegNet: Deeper layers produce smoother predictions and more category-tuned representations, with layer-4 top-1 activations using about 15% of features.
4. Experiments and Analysis
Experiments across CamVid, KITTI, and NYU evaluate SegNet on outdoor RGB and indoor RGBD semantic segmentation. SegNet performs strongly on small categories and overall accuracy, while dataset illumination, class ambiguity, and input resolution expose important boundaries.
- CamVid: SegNet achieves the highest class-average and global-average accuracy on CamVid, including challenging cars, pedestrians, and poles.The comparison includes methods using structure from motion, CRFs, dense depth maps, and temporal cues.
- NYU: SegNet improves over the same-input multi-scale convnet on 9 of 13 NYU classes.Another method achieves better accuracy using ground-plane detection and column-wise depth normalization.
- KITTI: SegNet performs better globally and comparably among classes on KITTI, although fence recognition benefits from temporal information.The fence class resembles buildings, making additional cues useful.
- Qualitative analysis: On CamVid day and dusk samples, SegNet retains small categories while producing smooth scene predictions, unlike CRF results that miss several important categories.Without structure-from-motion cues, SegNet can miss cars but fills the region with reasonable context-related classes.
- NYU: NYU RGBD predictions are largely correct, but category boundaries are not sharp because of low input resolution and interpolated depth near edges.These experiments omit ground-plane fitting, column-wise depth normalization, and multi-scale inputs.
- Transfer and training: CamVid pre-training transfers useful semantic cues to KITTI when the deepest layer is trained for two epochs, but direct pre-trained testing performs poorly under illumination differences.Training only a soft-max classifier performs poorly, whereas training layer 4 gives high-quality predictions with limited computation.
5. Conclusion
SegNet is a fully trainable feed-forward architecture that maps input images to semantic pixel labels, producing smooth and competitive predictions for outdoor and indoor scenes without CRF post-processing. Its broader flexibility includes transfer to other datasets, unsupervised training, and handling missing test-time input data.
- SegNet maps input images directly to pixel-wise semantic labels through a fully trainable feed-forward architecture.
- Deep feature encoding provides large spatial context, producing smoother segment labels than local patch-based classifiers.
- SegNet achieves competitive outdoor and indoor scene accuracy without CRF post-processing.
- Pre-trained SegNet performs well on other datasets with only a small extra computational effort.
- The encoder-decoder architecture can also be trained unsupervised and handle missing input data during testing.