PaperlayerView exact revision on arXiv

Source-linked AI summary

Stacked Deconvolutional Network for Semantic Segmentation

Jun Fu, Jing Liu, Yuhang Wang, Hanqing Lu

arXiv:1708.04943v1cs.CV

TL;DR

Semantic segmentation needs better recovery of spatial resolution after FCN downsampling. The paper proposes SDN, which stacks shallow deconvolutional units with dense connections and hierarchical supervision, and reports state-of-the-art performance across three datasets.

  • Problem

    FCN-based segmentation sacrifices spatial resolution through downsampling, limiting detailed localization and object delineation.

  • Method

    SDN stacks shallow deconvolutional units and uses intra-unit/inter-unit connections plus hierarchical supervision to integrate context, recover localization, and support optimization.

  • Results

    The best SDN model achieves state-of-the-art performance on PASCAL VOC 2012, CamVid, and GATECH.

  • Takeaways & Limitations

    The experiments and ablations show that stacking, connections, and hierarchical supervision benefit contextual capture, spatial recovery, and segmentation performance.

  • Takeaways & Limitations

    Stacking more units increases computational stress and GPU memory demand, while additional units beyond three bring only slight improvements.

Abstract

from arXiv · show

Recent progress in semantic segmentation has been driven by improving the spatial resolution under Fully Convolutional Networks (FCNs). To address this problem, we propose a Stacked Deconvolutional Network (SDN) for semantic segmentation. In SDN, multiple shallow deconvolutional networks, which are called as SDN units, are stacked one by one to integrate contextual information and guarantee the fine recovery of localization information. Meanwhile, inter-unit and intra-unit connections are designed to assist network training and enhance feature fusion since the connections improve the flow of information and gradient propagation throughout the network. Besides, hierarchical supervision is applied during the upsampling process of each SDN unit, which guarantees the discrimination of feature representations and benefits the network optimization. We carry out comprehensive experiments and achieve the new state-of-the-art results on three datasets, including PASCAL VOC 2012, CamVid, GATECH. In particular, our best model without CRF post-processing achieves an intersection-over-union score of 86.6% in the test set.

1 INTRODUCTION

Semantic segmentation must recover spatial detail lost by downsampling in FCNs. SDN addresses this with stacked shallow deconvolutional units, dense connections, and hierarchical supervision, achieving state-of-the-art results on three benchmarks.

  • Motivation: FCN-based segmentation gains semantic features from classification networks but sacrifices spatial resolution through downsampling.This loss limits detailed object delineation and boundary recovery.
  • Proposed approach: SDN stacks multiple shallow deconvolutional networks to capture contextual information while recovering localization information more finely.Each SDN unit combines an encoder downsampling process with a decoder upsampling process.
  • Network optimization: Dense intra-unit and inter-unit connections improve information and gradient propagation throughout the stacked network.Inter-unit connections also enable reuse of multi-scale information across units.
  • Network optimization: Hierarchical supervision during each unit’s upsampling process improves optimization and preserves discriminative feature representations.It provides additional gradient feedback to early layers and supervises pixel-wise labeling maps.
  • Evaluation: The proposed SDN achieves new state-of-the-art performance on PASCAL VOC 2012, CamVid, and GATECH.The experiments include comprehensive evaluations and component analyses.

2 RELATED WORK

Prior segmentation methods address contextual modeling or spatial recovery through dilation, pooling, unpooling, and deconvolution, but optimization and design limitations remain. SDN extends this line with stacked deconvolutional units, dense connections, and hierarchical supervision.

  • FCN limitations: FCNs use fully convolutional structures and bilinear interpolation for pixel-wise prediction, but can produce rough edges and object vanishing.These limitations motivate methods that preserve spatial information and refine delineation.
  • Context modeling: Dilated convolutions enlarge receptive fields and capture contextual information without losing resolution, yet resulting feature maps can remain coarse for object delineation.Related approaches also use hybrid or deformable convolutions to modify receptive fields.
  • Context modeling: Global pooling, atrous spatial pyramid pooling, and pyramid pooling collect multi-scale or global contextual features for segmentation.These methods differ in how they aggregate image-level or multi-scale information.
  • Spatial recovery: Unpooling and deconvolutional paths generate high-resolution predictions, while skip connections and refinement modules improve feature recovery and object delineation.Examples include OA-Seg, SegNet, U-Net, and RefineNet.
  • SDN positioning: SDN stacks shallow deconvolutional networks and adds dense connections and hierarchical supervision to make deeper deconvolutional models easier to optimize.The paper presents this as an expansion and redesign of the prior DCDN approach.

3 OUR APPROACH

SDN stacks shallow deconvolutional units with dense connections to capture context and progressively recover detailed, high-resolution segmentation features. Hierarchical supervision and skip connections support feature fusion and network optimization.

  • SDN architecture: SDN stacks multiple shallow encoder-decoder units to capture contextual information and progressively refine segmentation maps.Each unit uses two downsampling and two upsampling blocks, while the final prediction comes from the last unit's highest-resolution output.
  • Inference design: The highest-resolution output of the last SDN unit is used as the final inference prediction, with the unit resolution set to one-quarter of the input.The reduced resolution is designed to lower GPU memory usage and allow more SDN units to be stacked.
  • SDN architecture: Each SDN unit enlarges receptive fields through downsampling and restores resolution through repeated deconvolutional upsampling blocks.The encoder produces low-resolution features, while the decoder upsamples them toward one-quarter of the input resolution.
  • Dense connections: Dense intra-unit connections reuse features across convolutional layers, while compression layers reduce the channel growth and memory cost caused by concatenation.The densely connected structure supports multi-scale feature reuse, but its channel count grows linearly without compression.
  • Dense connections: Inter-unit skip connections propagate semantic information and gradients between adjacent units and fuse low-level features from the first unit into later units.These connections combine features across units and help restore refined object boundaries.
  • Hierarchical supervision: Hierarchical supervision maps intermediate upsampling features to pixel-wise labeling maps, applying supervision throughout the upsampling process.Score-map fusion combines same-resolution predictions before bilinear interpolation, and the resulting maps are trained with pixel-wise cross-entropy loss.

4 EXPERIMENTS

The paper evaluates SDN comprehensively on PASCAL VOC 2012, CamVid, and GATECH, including ablations of its components. The experiments report new state-of-the-art performance on all three datasets.

  • Experimental evaluation: SDN is evaluated on PASCAL VOC 2012, CamVid, and GATECH, with ablation studies examining component effects on PASCAL VOC 2012.The authors report comprehensive experiments across the three datasets.
  • Experimental evaluation: The reported experimental results show new state-of-the-art performance for SDN on all three evaluated datasets.The claim is based on the paper's comparative evaluation of the proposed framework.

4.1 Dataset and Evaluation Metrics

The experiments use PASCAL VOC 2012, CamVid, and GATECH, which cover object and street-scene segmentation settings with different class structures and dataset scales. Evaluation reports Mean IoU on PASCAL VOC 2012 and both Global Avg and Mean IoU on the other datasets.

  • Datasets: PASCAL VOC 2012 contains 20 foreground classes and one background class, with 10,582 augmented training images.The original split includes 1,464 training, 1,449 validation, and 1,456 test images.
  • Datasets: CamVid contains 367 training, 100 validation, and 233 test images at 360 × 480 resolution across 11 semantic categories.The dataset consists of five video sequences and has stronger spatial relationships among categories than PASCAL VOC 2012.
  • Datasets: GATECH contains 12,241 training frames and 7,071 testing frames from outdoor scenes labeled with eight semantic classes.Its classes include sky, ground, solid, porous, cars, humans, vertical mix, and main mix.
  • Evaluation metrics: Evaluation reports Mean IoU on PASCAL VOC 2012 and both Global Avg and Mean IoU on CamVid and GATECH.Global Avg measures correctly classified pixels over the whole dataset, while Mean IoU averages class-wise intersection-over-union ratios.

4.2 Results on PASCAL VOC 2012 dataset

On PASCAL VOC 2012, the experiments evaluate stacked SDN units and supporting design choices. Performance improves with deeper stacking, hierarchical supervision, score-map fusion, upsampling, multi-scale inference, and COCO pretraining, reaching 86.6% Mean IoU on the test set.

  • Stacking multiple SDN units: SDNM 1, SDNM 2, and SDNM 3 improve performance from 78.2% to 79.2% to 79.9% on the PASCAL VOC 2012 validation set.The authors also report improved object edges and discrimination as units increase.
  • Stacking multiple SDN units: Stacking SDN units creates a coarse-to-fine prediction process but increases computational stress and GPU memory demand.The authors therefore stack at most three units because additional units provide slight improvements.
  • Hierarchical supervision: At up ratio = 4, performance is 2.3 percentage points higher than at up ratio = 16, while additional hierarchical supervision yields smaller gains.SDNM 1 2 is 0.5 percentage points higher than SDNM 1 1, and supervision at all ratios increases performance from 78.0 to 78.2.
  • Score-map fusion: Score-map fusion improves SDNM 2 performance by 0.4 percentage points over SDNM 2− on the validation set.The comparison removes the score-map connection while keeping two SDN units.
  • Improvement strategies: COCO-pretrained SDNM 2∗ reaches 84.8% and exceeds COCO-pretrained Deeplabv3 at 82.7% by 2.1 percentage points.Adding an upsampling block gives 0.4% gain, and segmentation-map fusion adds another 1.2% improvement.
  • Comparison with state of the art: COCO-pretrained SDNM 3 achieves 86.6% Mean IoU on PASCAL VOC 2012 test data, outperforming RefineNet by 2.4%, PSPNet by 1.2%, and Deeplabv3 by 0.9%.Without COCO pretraining, SDNM 3 achieves 83.5% Mean IoU.

4.3 Results on CamVid dataset

On CamVid, SDN outperforms previous methods under both evaluated settings. VOC pretraining further improves both reported metrics, while the predicted maps capture multi-scale object appearance and spatial relationships.

  • Quantitative results: SDN without PASCAL VOC 2012 pretraining achieves 69.6% Mean IoU and 91.7% Global Avg on CamVid.The evaluation uses 367 training images and 233 test images.
  • Quantitative results: PASCAL VOC 2012 pretraining improves CamVid performance by 2.2 percentage points in Mean IoU and 1 percentage point in Global Avg.Nine of eleven object categories achieve best Mean IoU performance in the pretrained setting.
  • Comparison with prior methods: SDN outperforms previous methods under both CamVid settings, including methods that mine temporal information from video frames.The authors report a relatively large margin over those methods.
  • Qualitative results: Qualitative results show that SDN sketches multi-scale objects such as buildings and cars, as well as shaped objects such as poles and pedestrians.The authors associate these results with capturing contextual information and learning spatial relationships.

4.4 Results on GATECH dataset

On GATECH, SDN generalizes to a larger dataset with noisy annotations and outperforms prior methods under both pretrained and non-pretrained settings. SDN+ further improves mean IoU and Global Avg., including against methods using spatio-temporal information.

  • Both SDN settings outperform previous state-of-the-art methods on GATECH, with or without PASCAL VOC 2012 pretraining.
  • 53.5% mean IoU and 84.6% Global Avg. are achieved by SDN on GATECH.
  • 2.4% gain in Mean IoU and 1.7% gain in Global Avg. are obtained by SDN+ over SDN.
  • SDN outperforms methods exploiting spatio-temporal relationships between video frames despite using no temporal information.
  • Test images are presented with ground truth and predicted semantic maps for qualitative comparison.

5 CONCLUSION

The paper concludes that SDN combines stacked deconvolutional units, dense connections, and hierarchical supervision for semantic segmentation. Ablations associate these designs with contextual capture, spatial-resolution recovery, boundary localization, and improved benchmark performance.

  • 5 CONCLUSION: SDN stacks multiple units to create a deeper architecture and realize coarse-to-fine learning.
  • 5 CONCLUSION: Intra-unit and inter-unit connections together with hierarchical supervision are adopted to promote network optimization.
  • 5 CONCLUSION: Ablation experiments show that the designs capture contextual information and recover spatial resolution for accurate boundary localization.
  • 5 CONCLUSION: The best SDN model outperforms all previous works on three public benchmarks.
  • 5 CONCLUSION: Figure 7 shows GATECH test images alongside ground truth and predicted semantic maps.
Loading 1708.04943v1…