Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Liang-Chieh Chen; George Papandreou; Iasonas Kokkinos; Kevin Murphy; Alan L. Yuille

Source-linked AI summary

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L. Yuille

arXiv:1412.7062v4Dec 22, 2014cs.CVcs.LGcs.NE

TL;DR

Precise semantic segmentation requires localization that DCNNs do not provide well because their invariance favors high-level recognition. DeepLab combines DCNN responses with a fully connected CRF, achieving 71.6% mean IOU on the PASCAL VOC 2012 test set.

Problem
DCNNs recognize objects well but their invariance and large receptive fields limit precise localization for semantic segmentation.
Method
DeepLab couples DCNN recognition with a fully connected CRF to recover fine-grained object boundaries in semantic segmentation.
Results
71.6% mean IOU was achieved on the PASCAL VOC 2012 test set by the best multi-scale, large-FOV DeepLab model, the task’s best reported performance.
Takeaways & Limitations
Combining DCNNs with fully connected CRFs yields semantically accurate predictions and detailed segmentation maps while remaining computationally efficient.
Takeaways & Limitations
The authors identify full end-to-end integration and training of the CNN and CRF components as future work.

Abstract

from arXiv · show

Deep Convolutional Neural Networks (DCNNs) have recently shown state of the art performance in high level vision tasks, such as image classification and object detection. This work brings together methods from DCNNs and probabilistic graphical models for addressing the task of pixel-level classification (also called "semantic image segmentation"). We show that responses at the final layer of DCNNs are not sufficiently localized for accurate object segmentation. This is due to the very invariance properties that make DCNNs good for high level tasks. We overcome this poor localization property of deep networks by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF). Qualitatively, our "DeepLab" system is able to localize segment boundaries at a level of accuracy which is beyond previous methods. Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 71.6% IOU accuracy in the test set. We show how these results can be obtained efficiently: Careful network re-purposing and a novel application of the 'hole' algorithm from the wavelet community allow dense computation of neural net responses at 8 frames per second on a modern GPU.

1 INTRODUCTION

The introduction identifies downsampling and spatial insensitivity as key obstacles to applying DCNNs to semantic segmentation, and presents DeepLab’s atrous computation plus fully connected CRF as a fast, accurate, and simple solution.

Motivation: DCNNs’ transformation invariance enables strong high-level vision performance but inherently limits spatial accuracy for object-centric decisions.This motivates adding a mechanism that recovers fine spatial details for image labeling.
Challenges: Two technical hurdles affect DCNN-based image labeling: signal-resolution loss from pooling and striding, and spatial insensitivity caused by classifier invariance.The introduction describes these as distinct limitations of standard DCNNs when applied to pixel-level tasks.
Approach: DeepLab uses the atrous algorithm for dense DCNN computation and a fully connected CRF to capture fine details and combine classifier scores with low-level information.The CRF specifically boosts the model’s ability to capture fine details after DCNN predictions are produced.
Advantages: 8 fps is achieved by the atrous algorithm, while mean-field inference for the fully connected CRF requires 0.5 second.These figures are presented as DeepLab’s speed advantage in the introduction.
Advantages: 7.2% is DeepLab’s performance margin over Mostajabi et al. (2014), the second-best approach in the PASCAL semantic segmentation challenge.The introduction lists accuracy as one of DeepLab’s three main advantages.

2 RELATED WORK

The related work contrasts DeepLab’s direct pixel-level processing and combination of DCNN unary terms with pixel-level CRFs against two-stage, dense-labeling, and segmentation-free alternatives. It also situates the approach alongside concurrent work combining DCNNs with densely connected CRFs.

Pixel-level versus two-stage methods: DeepLab operates directly on pixels, unlike common two-stage methods that cascade bottom-up image segmentation with DCNN-based region classification.The two-stage design can make the system commit to potential front-end segmentation errors.
Dense image labeling: Prior dense-labeling methods apply DCNN features at multiple resolutions, smooth predictions with segmentation trees, or concatenate intermediate feature maps for pixel classification.These approaches include Farabet et al. (2013) and Hariharan et al. (2014a).
Segmentation-free techniques: Segmentation-free alternatives apply DCNNs to whole images in sliding-window fashion after replacing final fully connected layers with convolutional layers.Long et al. (2014) upsample and concatenate intermediate scores, whereas Eigen & Fergus (2014) refine predictions from coarse to fine.
DCNNs with CRFs: DeepLab’s main distinction is combining pixel-level CRFs with DCNN-based unary terms, rather than using CRFs only for proposals or superpixel-level local pairwise inference.The superpixel formulation can be limited by superpixel-computation errors and ignores long-range relationships.
Concurrent work: Bell et al. (2014) and Zheng et al. (2015) independently and concurrently pursued similar combinations of DCNNs and densely connected CRFs.Bell et al. focused on material classification, while Zheng et al. unrolled CRF mean-field inference steps.

3 CONVOLUTIONAL NEURAL NETWORKS FOR DENSE IMAGE LABELING

The section repurposes and fine-tunes ImageNet-pretrained VGG-16 into an efficient dense feature extractor for semantic segmentation. Dense 8-pixel-stride scoring, receptive-field control, and implementation optimizations enable accurate, fast spatial predictions.

Network adaptation: ImageNet-pretrained VGG-16 is repurposed and fine-tuned into an efficient dense feature extractor for semantic image segmentation.The network is adapted from a 16-layer classification model into a dense semantic segmentation system.
Dense score computation: Converting VGG-16’s fully connected layers into convolutional layers enables dense evaluation, while the hole algorithm reaches a target stride of 8 pixels instead of 32.The hole algorithm sparsely samples feature maps with input strides of 2 or 4 pixels while keeping filters intact.
Network adaptation: The 1000-way ImageNet classifier is replaced with a 21-way classifier, and training minimizes equally weighted cross-entropy terms across spatial output positions subsampled by 8.The loss is defined over each spatial position and label in the CNN output map.
Receptive-field control: Reducing the first fully connected layer spatially to 4×4 or 3×3 shrinks the receptive field to 128×128 or 308×308 pixels and cuts computation time 2−3 times.These dimensions correspond to zero-padded and convolutional modes, respectively.
Efficiency: 8 frames/sec are produced during testing from a 306×306 input, while training runs at 3 frames/sec; reducing fully connected channels from 4,096 to 1,024 further lowers computation and memory without sacrificing performance.The reported implementation uses Caffe and a Titan GPU.

4 DETAILED BOUNDARY RECOVERY: FULLY-CONNECTED CONDITIONAL RANDOM FIELDS AND MULTI-SCALE PREDICTION

The section addresses DCNN localization errors by coupling coarse recognition with a fully connected CRF that recovers fine-grained boundaries. It also explores multi-scale features and efficient mean-field inference to improve localization while keeping computation practical.

DCNN score maps identify objects and rough positions but struggle to pinpoint exact outlines because deep pooling increases invariance and receptive-field size.
The proposed alternative couples DCNN recognition with fully connected CRFs, producing accurate semantic segmentation and detailed object-boundary recovery.
Fully connected pairwise terms connect every pixel pair, using Potts compatibility with bilateral position-and-color and position-only Gaussian kernels.
0.5 sec average inference is reported for Pascal VOC images through mean-field updates implemented as Gaussian feature-space convolutions and accelerated by high-dimensional filtering.
The multi-scale model attaches two-layer MLPs to the input and first four max-pooling outputs, concatenating their feature maps with the network’s last-layer feature map.Each MLP uses 128 3x3 convolutional filters followed by 128 1x1 convolutional filters.

5 EXPERIMENTAL EVALUATION

The evaluation on PASCAL VOC 2012 shows that fully connected CRFs substantially improve DeepLab segmentation, especially around intricate object boundaries. Multi-scale features and larger fields of view provide additional gains, while the final models outperform competing methods on the test set.

Evaluation on Validation set: DeepLab-CRF improves validation performance by about 4% over DeepLab.The comparison uses the augmented PASCAL train set for training and the val set for evaluation.
Evaluation on Validation set: Fully connected CRFs significantly improve segmentation and enable accurate capture of intricate object boundaries.This improvement is shown qualitatively in comparisons between DeepLab and DeepLab-CRF.
Multi-Scale features: Multi-scale features improve DeepLab-MSc performance by about 1.5%, while adding a fully connected CRF yields about 4% further improvement.Multi-scale features also slightly refine object boundaries.
Field of View: DeepLab-CRF-7x7 reaches 67.64% validation performance with a 7×7 kernel and input stride 4, but runs at only 1.44 imageThe atrous algorithm controls the model’s field of view by adjusting input stride; the passage’s speed value is truncated after “1.44 image”.
Test set results: 66.4% and 67.1% mean IOU are achieved by DeepLab-CRF and DeepLab-MSc-CRF, respectively, on the official PASCAL VOC 2012 test set.Both models outperform TTI-Zoomout-16, FCN-8s, and MSRA-CFM.

6 DISCUSSION

The work combines deep convolutional neural networks with fully connected conditional random fields to produce semantically accurate, detailed, and computationally efficient segmentation. The authors identify future directions involving tighter integration, broader data, weak supervision, and further study of CNN–graphical-model interactions.

Contributions: The proposed CNN–CRF method produces semantically accurate predictions and detailed segmentation maps while remaining computationally efficient.It significantly advances state-of-the-art performance on the challenging PASCAL VOC 2012 semantic image segmentation task.
Future Work: The authors plan to integrate the CNN and CRF components end-to-end, evaluate more datasets and data sources, and pursue weakly supervised annotations.The proposed data sources include depth maps and videos; weak supervision includes bounding boxes or images, although the supplied passage is truncated.
Broader Perspective: The work lies at the intersection of convolutional neural networks and probabilistic graphical models, motivating further investigation of their interplay and synergistic potential.The intended application is solving challenging computer vision tasks.

PAPER REVISIONS

The paper revisions progressively improved DeepLab performance on the PASCAL VOC 2012 test set by adding multi-scale features and exploiting a large field of view, reaching 71.6%.

v1 Submission to ICLR 2015: 66.4%: The initial DeepLab-CRF model on the PASCAL VOC 2012 test set.This model was introduced in the v1 ICLR 2015 submission.
v2 Rebuttal for ICLR 2015: 67.1%: DeepLab-MSc-CRF incorporates multi-scale features from intermediate layers on the PASCAL VOC 2012 test set.This model was added in the ICLR 2015 rebuttal revision.
v3 Camera-ready for ICLR 2015: 71.6%: DeepLab-MSc-CRF-LargeFOV combines multi-scale features with a large field of view on the PASCAL VOC 2012 test set.The camera-ready revision reports 70.3% for DeepLab-CRF-LargeFOV and 71.6% when both enhancements are used.