Source-linked AI summary
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L. Yuille
TL;DR
Semantic segmentation needs accurate spatial predictions despite DCNNs’ tendency toward invariance and challenges from multiscale objects. DeepLab addresses this with atrous convolution, ASPP, and fully connected CRFs, advancing results across several challenging datasets.
Problem
Semantic segmentation requires spatially accurate predictions, but DCNN invariance can hamper localization and objects occur at multiple scales.
Method
DeepLab uses atrous convolution for dense feature extraction, ASPP for multiscale context, and fully connected CRFs for detailed boundary localization.
Results
DeepLab significantly advances state-of-the-art results across PASCAL VOC 2012, PASCAL-Context, PASCAL-Person-Part, and Cityscapes.
Takeaways & Limitations
The system provides a practical semantic-segmentation pipeline combining established DCNN and CRF modules with reported speed, accuracy, and simplicity advantages.
Takeaways & Limitations
The model can fail to capture delicate object boundaries when the unary predictions are insufficiently confident, even after CRF post-processing.
Abstract
from arXiv · showhide
In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First, we highlight convolution with upsampled filters, or 'atrous convolution', as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second, we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third, we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed "DeepLab" system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7% mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.
1 INTRODUCTION
The introduction identifies reduced feature resolution, multi-scale objects, and localization loss as central challenges for DCNN-based semantic segmentation. DeepLab addresses them with atrous convolution, multi-rate feature processing, and a fully connected CRF, while emphasizing speed and strong benchmark performance.
- Challenges: DeepLab targets three DCNN segmentation challenges: reduced feature resolution, objects at multiple scales, and reduced localization accuracy from DCNN invariance.These challenges motivate the system’s design and proposed solutions.
- Feature resolution: Atrous convolution removes late downsampling and inserts holes between filter taps, producing denser feature maps at higher sampling rates.The approach preserves larger-resolution feature responses while efficiently increasing sampling density.
- Multi-scale objects: DeepLab handles multi-scale objects by resampling a feature layer at multiple rates before convolution, using complementary effective fields of view to capture objects and context.This scheme is presented as more computationally efficient than processing multiple scaled images through all DCNN layers.
- Localization: A fully connected CRF improves fine-detail and boundary localization by complementing the spatially invariant DCNN responses.The introduction contrasts this approach with skip-layer hyper-column features for improving spatial accuracy.
- Practical advantages: 8 FPS is achieved by the dense DCNN on an NVidia Titan X GPU, while fully connected CRF mean-field inference requires 0.5 secs on a CPU.The introduction presents speed, benchmark accuracy, and simplicity as DeepLab’s practical advantages.
- System updates: The updated DeepLab supports multi-scale segmentation through multi-scale input processing or ASPP and includes a ResNet-based variant with better segmentation performance than the original system.These changes improve the system relative to its first conference-publication version.
2 RELATED WORK
Earlier semantic segmentation methods progressed from hand-crafted features and flat classifiers to DCNN-based region, dense-labeling, and segmentation-free approaches. DeepLab builds on fully convolutional DCNN work while using atrous convolution and fully connected CRF refinement as central ingredients.
- Earlier systems combined hand-crafted features with flat classifiers, while context and structured prediction improved performance but remained limited by feature expressiveness.
- The first DCNN family classified regions or superpixels produced by bottom-up segmentation, incorporating shape information while benefiting from sharp boundaries.
- A second family computed DCNN features densely and coupled them with independently obtained segmentations, using multi-resolution processing, segmentation trees, or skip layers.
- A third family directly produced dense category-level pixel labels with DCNNs, enabling segmentation-free approaches that transformed fully connected layers into convolutional layers.
- Most top-performing later methods adopted atrous convolution for efficient dense feature extraction, fully connected CRF refinement, or both.
- Atrous convolution originated in undecimated wavelet computation, was later called dilated convolution, and had been used for denser DCNN feature extraction before DeepLab.
3 METHODS
DeepLab uses atrous convolution to control feature-resolution density and enlarge filters’ receptive fields without added parameters or computation. It combines multirate atrous processing for scale robustness with fully connected CRFs for sharper object-boundary localization.
- Atrous Convolution: Atrous convolution computes feature responses at a desirable resolution and can be integrated during training or applied after network training.It avoids the severe spatial-resolution reduction caused by repeated max-pooling and striding in DCNNs.
- Atrous Convolution: Atrous convolution enlarges a k×k filter to ke = k + (k −1)(r −1) without increasing parameters or computation.The rate r inserts r −1 zeros between consecutive filter values, expanding the effective field of view.
- Atrous Spatial Pyramid Pooling: Atrous Spatial Pyramid Pooling applies parallel atrous convolutional layers with different sampling rates, processing and fusing features to handle scale variability.This approach explicitly accounts for objects at different scales using multirate features extracted from a single-scale representation.
- Fully Connected CRFs: Fully connected CRFs are coupled with DCNN recognition to provide fine-grained localization and recover object boundaries beyond the reach of existing methods.DCNN score maps can identify objects’ presence and rough position but generally produce smooth responses that do not delineate borders.
- Fully Connected CRFs: Because modern DCNN score maps are smooth and homogeneous, short-range CRFs can be detrimental when the goal is to recover detailed local structure.The method therefore combines DCNN outputs with contrast-sensitive potentials rather than further smoothing them.
4 EXPERIMENTAL RESULTS
Experiments evaluate DeepLab across four challenging semantic-segmentation datasets, showing that CRF post-processing, improved training and multi-scale designs, and deeper networks substantially improve performance. The final model reaches 79.7% on the PASCAL VOC 2012 test set, with additional gains reported on PASCAL-Context, PASCAL-Person-Part, and Cityscapes.
- Evaluation setup: DeepLab is evaluated on PASCAL VOC 2012, PASCAL-Context, PASCAL-Person-Part, and Cityscapes.PASCAL VOC performance is measured by pixel intersection-over-union averaged across 21 classes.
- PASCAL VOC 2012: 3–5% absolute mean IOU gains result from adding the fully connected CRF across model variants.The CRF also removes false positives and refines object boundaries in qualitative comparisons.
- PASCAL VOC 2012: 1.17% better performance is obtained with the “poly” learning-rate policy than with the “step” policy under matched training conditions.The poly policy uses power = 0.9 with the same batch size and training iterations.
- PASCAL VOC 2012: 79.7% test-set performance is achieved by the final best model on PASCAL VOC 2012, outperforming previous DeepLab variants.The model is reported as the top-performing method on the PASCAL VOC 2012 segmentation leaderboard.
- PASCAL-Context: 45.7% is achieved on PASCAL-Context after multi-scale processing, MS-COCO pretraining, ASPP, and dense CRF post-processing.This result outperforms the current state-of-the-art method by 2.4% without using its non-linear pairwise term.
- Cityscapes: 70.4% is obtained on the current Cityscapes test evaluation when training uses only the train set.An earlier pre-release benchmark reported 63.1%, or 64.8% with additional coarsely annotated images.
5 CONCLUSION
DeepLab repurposes image-classification networks for semantic segmentation using atrous convolution with upsampled filters. It extends this approach with atrous spatial pyramid pooling to encode objects and image context at multiple scales.
- 5 CONCLUSION: DeepLab applies atrous convolution with upsampled filters to repurpose image-classification networks for dense semantic-segmentation feature extraction.The method is designed for dense feature extraction in semantic segmentation.
- 5 CONCLUSION: Atrous spatial pyramid pooling encodes objects and image context at multiple scales.The extension captures both object information and surrounding image context across scales.