Source-linked AI summary
Fully Convolutional Networks for Semantic Segmentation
Jonathan Long, Evan Shelhamer, Trevor Darrell
TL;DR
Semantic segmentation requires pixelwise predictions that preserve both global semantics and local detail. The paper builds fully convolutional networks from pretrained classifiers, trains them end-to-end on whole images, and combines coarse and fine layers. These models achieve state-of-the-art results across multiple benchmarks while simplifying and speeding learning and inference.
Problem
Semantic segmentation must make a prediction at every pixel while reconciling global information about what an object is with local information about where it is.
Method
The paper reinterprets pretrained classification networks as fully convolutional models, fine-tunes them end-to-end, and combines deep coarse and shallow fine layers through a skip architecture.
Results
20% relative improvement in mean IU on PASCAL VOC 2012 is reported for FCN-8s, alongside state-of-the-art results and faster inference.
Takeaways & Limitations
Fully convolutional networks extend modern classification convnets to dense prediction while simplifying and speeding learning and inference.
Takeaways & Limitations
Shift-and-stitch trades denser output for filters that cannot access information at a finer scale than their original design.
Abstract
from arXiv · showhide
Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations by fine-tuning to the segmentation task. We then define a novel architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes one third of a second for a typical image.
1. Introduction
Semantic segmentation extends visual recognition to pixelwise labeling, but requires balancing global semantic understanding with local spatial detail. The paper addresses this with end-to-end fully convolutional networks and a multilayer architecture that combines coarse and fine information.
- Semantic segmentation assigns each pixel the class of its enclosing object or region, extending convolutional recognition from coarse to fine inference.
- Fully convolutional networks train end-to-end, pixels-to-pixels, and produce dense outputs for arbitrary-sized inputs without additional machinery.Dense feedforward computation, backpropagation, and in-network upsampling support whole-image learning and inference.
- The method transfers supervised classification representations to dense prediction by reinterpreting classification networks as fully convolutional and fine-tuning them for segmentation.
- Semantic segmentation must reconcile global information that resolves what with local information that resolves where.Deep feature hierarchies encode location and semantics across a local-to-global pyramid.
- The novel skip architecture combines deep, coarse semantic information with shallow, fine appearance information for detailed predictions.
- The paper reports state-of-the-art results on PASCAL VOC 2011-2, NYUDv2, and SIFT Flow.
2. Related work
Prior dense-prediction systems used convolutional methods alongside patchwise training, post-processing, and hybrid proposal pipelines. This work instead adapts pretrained classification architectures for direct, fully convolutional, end-to-end learning.
- Earlier work applied convolutional networks to detection, semantic segmentation, restoration, depth estimation, and other dense prediction tasks.
- Some hybrid proposal-classifier systems adapted classification networks to segmentation but were not learned end-to-end.
- Common prior techniques included small models, patchwise training, post-processing, shift-and-stitch output, multiscale processing, tanh nonlinearities, and ensembles.
- The paper adapts deep classification architectures, uses image classification as supervised pre-training, and fine-tunes fully convolutionally from whole-image inputs and ground truths.
- The authors directly compare their standalone, end-to-end FCN with prior semantic-segmentation results on PASCAL VOC and NYUDv2.
3. Fully convolutional networks
Fully convolutional networks reinterpret classification architectures as nonlinear filters that accept arbitrary-sized inputs and produce spatial outputs. Dense whole-image computation supports efficient learning and inference, while upsampling restores pixelwise detail from coarse predictions.
- FCN formulation: An FCN is a composition of convolutional-style layers that accepts inputs of any size and produces correspondingly sized, possibly resampled, spatial outputs.Its layers operate on three-dimensional feature arrays, with spatial locations linked to receptive fields in the input.
- Efficient dense computation: Whole-image stochastic gradient descent is equivalent to using all final-layer receptive fields as a minibatch when the loss sums over spatial output locations.Overlapping receptive fields make layer-by-layer feedforward and backpropagation much more efficient than independent patch processing.
- Adapting classifiers: Convolutionalizing fully connected layers converts fixed-input classification networks into models that output classification maps for arbitrary-sized inputs.The resulting maps are equivalent to evaluating the original network over overlapping patches, but share computation across those regions.
- Efficient dense computation: 22 ms produces a 10 × 10 output grid from a 500 × 500 image, making the fully convolutional AlexNet over 5 times faster than naive patchwise evaluation.The corresponding backward pass also benefits from dense computation: 37 ms for the map versus 2.4 ms for a single-image backward pass.
- Recovering dense outputs: Subsampling coarsens FCN outputs by the receptive-field pixel stride, so arbitrary-size prediction is not automatically pixel-resolution prediction.This output-resolution limitation motivates connecting coarse outputs back to dense pixels through shift-and-stitch, interpolation, or learned upsampling.
- Recovering dense outputs: Shift-and-stitch densifies outputs without interpolation but trades away access to finer-scale information, whereas learned in-network upsampling is reported as fast and effective.The authors do not use shift-and-stitch, finding learned upsampling more effective and efficient, especially with skip-layer fusion.
4. Segmentation Architecture
The architecture converts classification networks into fully convolutional segmentation models, then refines coarse predictions by fusing deep semantic and shallow spatial information. End-to-end skip architectures improve detail and validation performance, while extra data further raises the score.
- From classifier to dense FCN: Classification networks are converted into fully convolutional models with in-network upsampling, pixelwise loss, and fine-tuning for dense segmentation.The adapted models include AlexNet, VGG16, and GoogLeNet variants.
- From classifier to dense FCN: The baseline FCN-VGG16 achieves 56.0 mean IU on the PASCAL VOC 2011 validation set, compared with 52.6 on test.Fine-tuning from classification produces reasonable segmentation predictions across the evaluated networks.
- Combining what and where: Skip links address coarse 32-pixel-stride outputs by combining final predictions with lower layers that have finer strides.The resulting directed acyclic graph combines local appearance information with global semantic structure.
- Combining what and where: FCN-16s adds stride-16 predictions from pool4, upsamples them, and sums them with stride-32 conv7 predictions before returning to image resolution.The added parameters are initialized so the network begins with the coarser model’s predictions and is then learned end-to-end.
- Combining what and where: 3.0 mean IU raises validation performance to 62.4 after learning the FCN-16s skip architecture.The improvement is accompanied by finer output structure, whereas learning from pool4 alone or only lowering the learning rate performs poorly or insignificantly.
- Combining what and where: 62.7 mean IU is obtained after further fusion, with only slight improvements in output smoothness and detail before diminishing returns halt fusion at lower layers.The metric emphasizes large-scale correctness, while the visible detail gains also become small.
5. Results
The FCN is evaluated across semantic segmentation, scene parsing, RGB-D input, and joint semantic-geometric prediction, achieving state-of-the-art results with substantially reduced inference time.
- Experimental scope: FCN models were evaluated on PASCAL VOC, NYUDv2, and SIFT Flow as pixel-prediction tasks.The experiments also extended the model to multimodal RGB-D input and multitask prediction.
- PASCAL VOC: 20% relative improvement over the state-of-the-art mean IU was achieved on PASCAL VOC 2011 and 2012 test sets.Inference time was reduced 114× for the convnet alone and 286× overall.
- NYUDv2: NYUDv2 experiments compared RGB, RGB-D, and fusion variants on the standard split of 795 training and 654 testing images.The RGB-D model used four-channel early fusion, while other variants incorporated depth embeddings and late fusion.
- SIFT Flow: On SIFT Flow, a two-headed FCN performed as well on semantic and geometric prediction as two independently trained models.Learning and inference were essentially as fast as each independent model, and results showed state-of-the-art performance on both tasks.
- Qualitative results: The PASCAL VOC qualitative results show recovery of fine structures, separation of closely interacting objects, and robustness to occluders.The figure also includes a failure case involving lifejackets and a boa.
6. Conclusion
The paper concludes that fully convolutional networks generalize classification convnets to segmentation, while multilayer combinations improve results and simplify and speed training and inference.
- Fully convolutional networks form a rich model class that includes modern classification convnets as a special case.
- Extending classification networks to segmentation and combining multiple resolutions improves state-of-the-art performance while simplifying and speeding learning and inference.
A. Upper Bounds on IU
The paper uses downsampled-and-upsamped ground truth to estimate mean IU upper bounds at different prediction scales, showing that high mean IU does not require pixel-perfect outputs.
- Approximate mean IU upper bounds are computed by downsampling ground-truth images and upsampling them at various scales.
- Pixel-perfect prediction is unnecessary for mean IU well above the state-of-the-art, while mean IU is not a good measure of fine-scale accuracy.
B. More Results
On PASCAL-Context, FCN-8s improves over the previous state of the art for the 59-class semantic segmentation task.
- 35.1 mean IU represents an 11% relative improvement over the previous state-of-the-art on PASCAL-Context.The evaluation follows the 59-class task defined by prior work, using training and validation sets.
Changelog
The arXiv version records ongoing corrections and added material. Version 2 adds appendices and corrects evaluation numbers, a metric, a formula, and timing information.
- Version 2 adds Appendix A on upper bounds for mean IU and Appendix B with PASCAL-Context results.
- Version 2 corrects PASCAL validation numbers after some validation images were previously included in training.
- Version 2 corrects SIFT Flow mean IU because the earlier evaluation used an inappropriately strict metric.
- Version 2 corrects an error in the frequency weighted mean IU formula and updates timing numbers for an improved implementation.