Source-linked AI summary
Learning Deconvolution Network for Semantic Segmentation
Hyeonwoo Noh, Seunghoon Hong, Bohyung Han
TL;DR
Semantic segmentation methods based on fully convolutional networks can struggle with object-scale variation and detailed structures. This paper learns a deconvolution network for proposal-wise segmentation and achieves state-of-the-art PASCAL VOC 2012 performance among methods trained without external data.
Problem
Fully convolutional segmentation methods can mishandle objects at scales outside their fixed receptive field and lose detailed structures through coarse label maps and simple upsampling.
Method
The method learns a multilayer deconvolution network with unpooling and applies it to candidate proposals before aggregating pixel-wise predictions across the image.
Results
The ensemble achieves the best accuracy among methods trained only on PASCAL VOC 2012 and demonstrates state-of-the-art performance on its segmentation benchmark.
Takeaways & Limitations
Proposal-wise deconvolution segmentation captures fine object details and handles scale variation, while ensembling with FCN exploits complementary strengths.
Takeaways & Limitations
Fully convolutional approaches may still provide stronger image context than instance-wise prediction, motivating the ensemble’s use of both methods.
Abstract
from arXiv · showhide
We propose a novel semantic segmentation algorithm by learning a deconvolution network. We learn the network on top of the convolutional layers adopted from VGG 16-layer net. The deconvolution network is composed of deconvolution and unpooling layers, which identify pixel-wise class labels and predict segmentation masks. We apply the trained network to each proposal in an input image, and construct the final semantic segmentation map by combining the results from all proposals in a simple manner. The proposed algorithm mitigates the limitations of the existing methods based on fully convolutional networks by integrating deep deconvolution network and proposal-wise prediction; our segmentation method typically identifies detailed structures and handles objects in multiple scales naturally. Our network demonstrates outstanding performance in PASCAL VOC 2012 dataset, and we achieve the best accuracy (72.5%) among the methods trained with no external data through ensemble with the fully convolutional network.
1. Introduction
The introduction identifies scale handling and loss of object detail as key FCN limitations, then presents a learned multi-layer deconvolution network applied to object proposals for finer, scale-robust semantic segmentation. The method achieves outstanding PASCAL VOC 2012 performance, with its ensemble with FCN methods attaining the best accuracy.
- Limitations: FCN-based segmentation struggles with objects at scales that differ from its fixed receptive field, causing fragmentation or mislabeling.Its fixed-size receptive field supports only a single scale of semantics within an image.
- Limitations: FCN methods also lose fine object structures because they upsample a coarse label map using an overly simple deconvolution procedure.The original FCN uses a 16 × 16 label map and bilinear interpolation to recover the input image size.
- Method contributions: The proposed approach learns a multi-layer deconvolution network composed of deconvolution, unpooling, and ReLU layers.The introduction presents this as a previously unattempted approach to semantic segmentation, to the authors’ knowledge.
- Method contributions: Applying the trained network to individual object proposals yields instance-wise segmentations that are combined into a final map, avoiding FCN scale issues and preserving finer details.Proposal-wise prediction is designed to handle objects across scales naturally.
- Results: The deconvolution network achieves outstanding performance when trained only on PASCAL VOC 2012, while ensembling with [17] obtains the best accuracy.The ensemble exploits heterogeneous and complementary characteristics relative to FCN-based methods.
2. Related Work
Prior semantic segmentation work includes classification-based methods, fully convolutional networks, weakly supervised techniques, and deconvolution networks developed mainly for reconstruction and visualization. Classification methods use superpixels or region proposals, while FCNs produce coarse score maps through convolutional processing.
- Classification-based methods: Classification-based methods label multi-scale superpixels or region proposals, then combine or refine predictions for pixel-wise segmentation.Mostajabi et al. and Farabet et al. classify multi-scale superpixels; other methods classify region proposals and refine image-level labels.
- Fully convolutional networks: FCNs reinterpret fully connected layers as convolutions and generate segmentation from coarse class score maps obtained by feedforwarding an input image.The approach uses large receptive fields, a simple interpolation filter for deconvolution, and fine-tunes only the CNN portion.
- Weakly supervised methods: Weakly supervised methods refine bounding-box annotations iteratively or perform segmentation from image-level annotations using multiple instance learning.These settings use less supervision than methods trained with full segmentation labels.
- Deconvolution networks: Learned deconvolution networks have primarily reconstructed input images and visualized activated features, using unpooling to reverse max-pooling effects.The unpooling operation stores pooled locations to support reconstruction from feature representations.
3. System Architecture
The system combines a VGG 16-layer convolutional feature extractor with a mirrored deconvolution network that reconstructs dense, class-specific segmentation maps. It performs instance-wise prediction on candidate proposals and aggregates them into a whole-image semantic segmentation map.
- Network architecture: The network pairs a VGG 16-layer convolutional feature extractor with a mirrored deconvolution network that generates object segmentation from extracted features.The convolutional component uses 13 convolutional layers and two fully connected layers, while the deconvolutional component contains repeated unpooling, deconvolution, and rectification layers.
- Unpooling and deconvolution: Unpooling restores activation-map size using recorded pooling locations, while learned deconvolution filters densify sparse activations through convolution-like operations.Deconvolutional filters associate one input activation with multiple outputs and learn bases for reconstructing object shapes.
- Shape reconstruction: Hierarchical deconvolutional layers reconstruct coarse-to-fine, class-specific object structures, enabling dense pixel-wise probability maps and detailed segmentation.Lower layers capture overall object shape, while higher layers encode class-specific fine details and suppress irrelevant activations.
- Proposal-wise segmentation: The algorithm treats semantic segmentation as instance-wise segmentation, applying the network to each candidate proposal and aggregating proposal outputs in the original image space.This proposal-wise formulation handles objects at various scales and identifies fine object details while reducing prediction search space and training memory requirements.
4. Training
Training addresses the network’s depth and limited data through batch normalization and a two-stage curriculum. The network first learns from centered ground-truth object crops, then is fine-tuned on more challenging object proposals.
- Training challenges: The network is twice deeper than [22] and has many parameters, while only 12031 PASCAL training and validation images are available.This mismatch makes training a deep semantic-segmentation network difficult.
- Batch normalization: Batch normalization is added after every convolutional and deconvolutional layer to normalize inputs to a standard Gaussian distribution.It reduces internal-covariate shift during optimization.
- Batch normalization: Without batch normalization, the network reaches a poor local optimum, making normalization critical for optimization.The method uses batch normalization to help escape local optima.
- Two-stage training: A two-stage training method first trains on easy examples and then fine-tunes the network with more challenging examples.This addresses the large semantic-segmentation search space relative to the number of training examples.
- Two-stage training: The first stage uses centered ground-truth object crops to limit variation in object location and size, while the second stage uses object proposals for harder examples.Reducing the search space enables successful training with fewer examples before introducing proposal-based variation.
5. Inference
Inference applies the trained deconvolution network to candidate proposals, aggregates their outputs into an image-level segmentation, and optionally ensembles the result with FCN. Pixel-wise score aggregation, softmax normalization, and fully connected CRF refinement produce the final labeling, while the two networks provide complementary strengths.
- Proposal-wise inference: The network segments individual proposals, then aggregates all proposal outputs to produce whole-image semantic segmentation.An ensemble with FCN can optionally further improve performance.
- Proposal-wise inference: Pixel-wise maximum or average aggregation suppresses noise from misaligned proposals and cluttered backgrounds.These operations are described as sufficiently effective for robust results.
- Proposal-wise inference: Proposal score maps are zero-padded into image space, aggregated by class, converted with softmax, and refined by a fully connected CRF for final pixel-wise labeling.The CRF unary potentials are obtained from the resulting pixel-wise class-conditional probability maps.
- Deconvolution–FCN combination: The deconvolution network captures fine object details and handles varying scales through instance-wise prediction, complementing FCN shape and context extraction.FCN can be advantageous for capturing image context at coarse scales.
- Deconvolution–FCN combination: The ensemble averages independently computed class-conditional probability maps from the deconvolution network and FCN before applying CRF.This combines the outputs of both algorithms into the final semantic segmentation.
6. Experiments
Experiments evaluate DeconvNet on PASCAL VOC 2012 using IoU-based scoring and proposal-wise inference. The method provides fine, multi-scale segmentations, while ensemble evaluation further improves mean IoU over FCN-8s and DeconvNet.
- Dataset: Only PASCAL VOC 2012 images are used for training, unlike some state-of-the-art methods that use additional data.Training uses augmented segmentation annotations with all training and validation images, and performance is evaluated on test images.
- Inference: Approximately 2000 edge-box proposals are generated per test image, the top 50 are selected, and pixel-wise maxima aggregate proposal predictions.The selected proposal count is reported as sufficient for accurate segmentation in practice.
- Evaluation Setup: DeconvNet is evaluated on PASCAL VOC 2012 using the comp6 protocol and Intersection over Union (IoU) scores across 20 object categories.The benchmark contains 1456 test images.
- Quantitative Results: 10.3% mean IoU improvement is obtained by ensembling with FCN-8s relative to FCN-8s, while the improvement relative to DeconvNet is 3.1% point.CRF post-processing additionally enhances accuracy by approximately 1% point.
- Instance-wise Prediction: Aggregating proposals in decreasing size order identifies progressively finer object structures, typically captured by small proposals.This supports instance-wise prediction for accurate segmentation and effective handling of multi-scale objects.
- Qualitative Results: DeconvNet produces finer segmentations than FCN and handles multi-scale objects effectively, although proposal misalignment or background proposals can yield noisy predictions.FCN tends to fail on overly large or small objects because of its fixed-size receptive field.
7. Conclusion
The paper proposes a learned deconvolution network for dense, precise object segmentation through progressive reconstruction. Instance-wise prediction handles scale variation, while ensembling with FCN outputs improves accuracy.
- Contribution: The proposed method learns a deconvolution network for semantic segmentation and generates dense, precise object segmentation masks.The network progressively reconstructs coarse-to-fine object structures through a sequence of deconvolution operations.
- Contribution: Instance-wise prediction handles object-scale variation by avoiding the fixed-size receptive-field limitation of fully convolutional networks.The method applies prediction per object instance, making segmentation less dependent on a single receptive-field scale.
- Contribution: The proposed method and FCN have complementary segmentation characteristics, so combining their outputs through an ensemble improves accuracy.The conclusion identifies ensembling as a further proposed approach; Figure 7 notes that CRF removes some noise without significantly improving quantitative performance.