PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Convolutional Networks for Semantic Segmentation

Evan Shelhamer, Jonathan Long, Trevor Darrell

arXiv:1605.06211v1cs.CV

TL;DR

Semantic segmentation must combine semantic recognition with precise spatial localization, motivating models that produce dense pixelwise predictions efficiently. The paper converts pretrained classification networks into fully convolutional, end-to-end segmentation models and adds skip connections that fuse coarse semantic and fine appearance features. The resulting FCNs improve prior segmentation results across multiple benchmarks, including a 30% relative mean-IU improvement on PASCAL VOC 2012, while retaining fast inference.

  • Problem

    Semantic segmentation requires global information to determine what objects are and local information to determine where they are, raising whether classification networks can support accurate pixelwise recognition.

  • Method

    The paper reinterprets pretrained classification networks as fully convolutional models, fine-tunes them end-to-end on whole images, and fuses deep and shallow features with skip connections.

  • Results

    The FCN achieves a 30% relative improvement in mean IU over the previous best on PASCAL VOC, with inference reduced by 114× for the convnet alone or 286× overall.

  • Takeaways & Limitations

    Fully convolutional networks provide an efficient framework for dense pixelwise prediction by combining pretrained representations, cross-layer feature fusion, and end-to-end whole-image learning.

  • Takeaways & Limitations

    Subsampling makes fully convolutional classifier outputs coarse, while denser alternatives trade receptive-field access or computational efficiency against spatial detail.

Abstract

from arXiv · show

Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, improve on the previous best result in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations by fine-tuning to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves improved segmentation of PASCAL VOC (30% relative improvement to 67.2% mean IU on 2012), NYUDv2, SIFT Flow, and PASCAL-Context, while inference takes one tenth of a second for a typical image.

1 INTRODUCTION

The paper addresses the tension between semantic recognition and precise localization by training fully convolutional networks end-to-end for dense pixelwise prediction. It adapts pretrained classification networks and fuses deep semantic features with shallow appearance features to improve segmentation.

  • 1 INTRODUCTION: FCNs trained end-to-end, pixels-to-pixels, exceed previous semantic-segmentation results without additional machinery.They perform whole-image dense feedforward computation and backpropagation, using in-network upsampling for pixelwise prediction with subsampling.
  • 1 INTRODUCTION: Pretrained classification networks are reinterpreted as fully convolutional and fine-tuned to transfer learned representations to dense prediction.This transfers recent classification success to segmentation without relying on the small, unpretrained convnets used in some prior work.
  • 1 INTRODUCTION: Semantic segmentation must reconcile global information for what with local information for where.The paper asks how local decisions can respect global structure and whether classification representations suffice for accurate pixelwise recognition.
  • 1 INTRODUCTION: A skip architecture fuses deep, coarse semantic information with shallow, fine appearance information for more detailed predictions.The feature hierarchy is treated as a nonlinear local-to-global pyramid.
  • 1 INTRODUCTION: The work improves and extends earlier FCN results through further tuning, ablations, and analyses of context, shape, and unified object-scene pixelwise prediction.The journal version also learns skip architectures all at once rather than in stages.

2 RELATED WORK

The paper situates FCNs among prior dense-prediction and fully convolutional approaches, then distinguishes its end-to-end transfer-learning formulation and learned cross-layer feature fusion. Earlier methods commonly used smaller models, patchwise training, refinement, or multiscale machinery, while later work extended FCNs to additional tasks and improved their segmentation accuracy.

  • 2 RELATED WORK: FCNs are related to earlier fully convolutional inference and learning systems for detection, segmentation, restoration, and pose estimation.Historical examples include Matan et al., Wolf and Platt, and Tompson et al..
  • 2 RELATED WORK: Prior dense-prediction approaches commonly used small models, patchwise training, refinement, interlacing, multiscale pyramids, saturating nonlinearities, or ensembles.These design patterns appear across semantic segmentation, boundary prediction, restoration, and depth-estimation systems.
  • 2 RELATED WORK: The method adapts deep classification architectures with supervised pre-training and fine-tunes them fully convolutionally from whole images and ground truths.Unlike hybrid proposal-classifier systems such as those of Hariharan et al. and Gupta et al., it is learned end-to-end.
  • 2 RELATED WORK: The paper fuses features across layers into a nonlinear local-to-global representation tuned end-to-end.Related fusion methods include Sermanet et al., Hariharan et al., and Mostajabi et al., but they do not learn end-to-end in the same way.
  • 2 RELATED WORK: Subsequent FCN extensions address region proposals, contour detection, depth regression, optical flow, and weakly supervised semantic segmentation.Later segmentation systems improved FCNs using dilated convolution, dense CRF inference, feature normalization, global pooling, learned deconvolution, unpooling, and skip layers.

3 FULLY CONVOLUTIONAL NETWORKS

Fully convolutional networks reinterpret classification architectures as nonlinear filters that accept arbitrary-sized inputs and produce spatial outputs, enabling efficient whole-image dense prediction. Their coarse outputs can be densified through interpolation or shift-and-stitch, while whole-image training exploits overlapping receptive fields for efficient learning.

  • 3 FULLY CONVOLUTIONAL NETWORKS: Fully convolutional networks accept inputs of any size and produce correspondingly sized, possibly resampled, spatial outputs.They consist only of locally defined convolution, pooling, and activation-style layers, forming a nonlinear filter.
  • 3 FULLY CONVOLUTIONAL NETWORKS: Whole-image stochastic gradient descent aggregates spatial loss and parameter gradients across all output receptive fields, matching patchwise updates while avoiding redundant computation.When receptive fields overlap substantially, layer-by-layer whole-image computation is more efficient than independent patch processing.
  • 3.1 Adapting classifiers for dense prediction: Converting fully connected layers into convolutions transforms fixed-dimensional classifiers into spatial-output networks while amortizing computation across overlapping input regions.The resulting spatial maps support straightforward forward and backward passes with convolutional efficiency.
  • 3.1 Adapting classifiers for dense prediction: 22 ms produces a 10 × 10 output grid from a 500 × 500 image, more than 5 times faster than naïve patch evaluation.The equivalent AlexNet single-image classification inference takes 1.2 ms for a 227 × 227 input.
  • 3.1 Adapting classifiers for dense prediction: Subsampling makes classifier-derived output maps coarser than the input by the receptive-field pixel stride, limiting their spatial precision.This reduction motivates connecting coarse outputs back to pixels through interpolation or shifted-input reconstruction.
  • 3.2 Shift-and-stitch is filter dilation: Shift-and-stitch obtains dense predictions by processing every input shift within the downsampling factor and interlacing outputs, with filter dilation reproducing the result efficiently.The dilation tradeoff preserves receptive-field size and denser output but prevents filters from accessing information at a finer scale than their original design; the authors instead use learned upsampling in their model.
  • 3.3 Upsampling is convolutional: Whole-image training is effective and efficient, whereas sampling-based patchwise training does not yield faster or better convergence for dense prediction.The paper also describes interpolation as another way to connect coarse outputs to dense pixels.

4 SEGMENTATION ARCHITECTURE

The segmentation architecture converts pretrained classification networks into fully convolutional, end-to-end dense predictors, then fuses coarse semantic and fine appearance features through learned skip connections. FCN-VGG16 provides the base network, while progressively finer fusion improves validation mean IU from 65.0 with FCN-16s to 65.5 with FCN-8s.

  • 4.1 From classifier to dense FCN: The architecture converts pretrained classification networks into fully convolutional segmenters with in-network upsampling, pixelwise loss, and end-to-end fine-tuning.AlexNet, VGG, and GoogLeNet are adapted by replacing classifiers with convolutions and transferring learned representations.
  • 4.1 From classifier to dense FCN: FCN-VGG16 reaches 56.0 mean IU on the PASCAL VOC 2011 validation set, outperforming the prior 52.6 test result used for comparison.FCN-VGG16 is selected as the base network because it outperforms the adapted alternatives in this setting.
  • 4.2 Image-to-image learning: The method trains on whole images with image-to-image optimization, and properly tuned online learning achieves higher accuracy and faster convergence than accumulated-gradient batching.The optimization uses a spatially summed loss, so learning rates and batch regimes require specific tuning.
  • 4.3 Combining what and where: Skip connections fuse shallow fine-stride features with deep coarse semantic features, turning the network into a jointly learned multi-stream DAG that improves spatial precision.The streams are aligned by upsampling and cropping before their predictions are fused; shallower layers provide more local information.
  • 4.3 Combining what and where: 65.0 mean IU with FCN-16s rises to 65.5 with FCN-8s, after which fusion improvements show diminishing returns.FCN-16s adds pool4 predictions at stride 16, while FCN-8s further incorporates pool3 at stride 8.

5 RESULTS

The FCN skip architecture improves semantic segmentation across PASCAL VOC, NYUDv2, SIFT Flow, and PASCAL-Context, with gains over prior methods and efficient inference.

  • PASCAL VOC: 30% relative improvement in mean IU over the previous best is achieved on PASCAL VOC 2011/2012, with substantially faster inference.The reported inference reduction is 114× for the convolutional network alone or 286× overall.
  • NYUDv2: The late-fusion RGB-HHA network is the most accurate NYUDv2 variant, while early RGB-D fusion provides little benefit.
  • SIFT Flow: On SIFT Flow, a two-headed FCN jointly predicts semantic and geometric labels and performs better on both tasks.The evaluation uses the standard split of 2,488 training and 200 test images.
  • PASCAL-Context: 39.1 mean IU on the 59-class PASCAL-Context task yields a relative improvement of more than 10% over the previous best result.
  • Qualitative results: FCN outputs recover fine structures, separate closely interacting objects, and remain robust to occluders, but confuse lifejackets with people and hair with dogs.

6 ANALYSIS

The analysis studies context, shape, background modeling, optimization, and output resolution to explain FCN behavior and limits.

  • 6.1 Cues: Foreground masking during inference is catastrophic, whereas training with masked foregrounds still permits object-segment recognition, indicating that FCNs incorporate context while relying on foreground pixels.
  • 6.1 Cues: Shape-only input produces lower accuracy than foreground-masked input but remains surprisingly accurate, showing that FCNs can learn from shape and context alone.
  • 6.1 Cues: Removing the learned background model lowers the score by 1 point, suggesting that a dedicated background model is not vital despite reducing parameters by less than 0.1%.
  • 6.2 Momentum and batch size: Heavy online learning with high momentum trains more accurate models in less wall-clock time, supported by an approximate momentum–batch-size tradeoff.
  • 6.3 Upper bounds on IU: Pixel-perfect prediction is unnecessary for mean IU well above state of the art, and recognition rather than resolution is the bottleneck at every tested stride.

7 CONCLUSION

FCNs improve pixelwise prediction by transferring classifier weights, fusing representations across layers, and learning end-to-end on whole images.

  • FCNs address pixelwise tasks by transferring pre-trained classifier weights, fusing layer representations, and learning end-to-end on whole images.End-to-end pixel-to-pixel operation simplifies and speeds learning and inference.
Loading 1605.06211v1…