PaperlayerView exact revision on arXiv

Source-linked AI summary

ShelfNet for Fast Semantic Segmentation

Juntang Zhuang, Junlin Yang, Lin Gu, Nicha Dvornek

arXiv:1811.11254v6cs.CV

TL;DR

Semantic segmentation models can impose heavy computation costs, while methods aimed at speed may not maintain accuracy. ShelfNet uses multiple encoder-decoder branches with cross-scale skip connections and shared-weight blocks. The paper reports accurate segmentation with faster inference than offline models and competitive real-time performance.

  • Problem

    Deep segmentation models can require heavy computation, while pruning and distillation are typically insufficiently fast and smaller models can lose accuracy.

    Source: §1 · p. 1

  • Method

    ShelfNet uses multiple encoder-decoder pairs with skip connections at each spatial level and shares convolution weights within residual blocks.

    Source: §1 · pp. 1–2

  • Results

    ShelfNet reports comparable accuracy to state-of-the-art offline models at 4–5 times faster inference and the highest Cityscapes mIoU among real-time models at speed comparable to BiSeNet.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The reported results support ShelfNet as an accurate, fast segmentation architecture evaluated on three benchmark datasets.

    Source: §5 · p. 8

Abstract

from arXiv · show

In this paper, we present ShelfNet, a novel architecture for accurate fast semantic segmentation. Different from the single encoder-decoder structure, ShelfNet has multiple encoder-decoder branch pairs with skip connections at each spatial level, which looks like a shelf with multiple columns. The shelf-shaped structure can be viewed as an ensemble of multiple deep and shallow paths, thus improving accuracy. We significantly reduce computation burden by reducing channel number, at the same time achieving high accuracy with this unique structure. In addition, we propose a shared-weight strategy in the residual block which reduces parameter number without sacrificing performance. Compared with popular non real-time methods such as PSPNet, our ShelfNet achieves 4$\times$ faster inference speed with similar accuracy on PASCAL VOC dataset. Compared with real-time segmentation models such as BiSeNet, our model achieves higher accuracy at comparable speed on the Cityscapes Dataset, enabling the application in speed-demanding tasks such as street-scene understanding for autonomous driving. Furthermore, our ShelfNet achieves 79.0\% mIoU on Cityscapes Dataset with ResNet34 backbone, outperforming PSPNet and BiSeNet with large backbones such as ResNet101. Through extensive experiments, we validated the superior performance of ShelfNet. We provide link to the implementation \url{https://github.com/juntang-zhuang/ShelfNet-lw-cityscapes}.

1. Introduction

ShelfNet addresses the high computation burden of semantic segmentation with a multi-branch encoder-decoder architecture and shared convolution weights. The paper reports strong accuracy and speed across benchmarks, including Cityscapes comparisons with BiSeNet and larger-backbone models.

1. Introduction

  • Deep segmentation models impose heavy computation costs, while pruning and distillation are described as typically too slow and smaller models as losing accuracy.These constraints motivate a new architecture for fast segmentation that maintains satisfying accuracy.Source: §1 · p. 1
  • ShelfNet replaces the common single encoder-decoder with multiple encoder-decoder pairs and skip connections at each spatial level to improve information flow and segmentation accuracy.The architecture draws multi-scale features from a CNN backbone into a multi-column segmentation shelf.Source: §1 · p. 1, Figure 1 · p. 2
  • Sharing the two convolution layers' weights within a residual block reduces parameter count without sacrificing accuracy.The paper proposes this shared-weight strategy for blocks in the segmentation shelf.Source: §1 · pp. 1–2
  • 75.8% mIoU with ResNet18 is reported at speed comparable to BiSeNet, while ResNet34 ShelfNet reaches 79.0% mIoU and outperforms PSPNet and BiSeNet with larger backbones such as ResNet101.Both comparisons are on Cityscapes.Source: §1 · p. 2

2. Related Work

Related work describes encoder-decoder and dilated-CNN approaches, then reviews real-time models that trade accuracy, speed, or feature handling. ShelfNet is presented as a multi-pair alternative with cross-scale skip connections.

2.1. Semantic Segmentation

  • Encoder-decoder models such as U-Net, RefineNet, and SegNet downsample spatial resolution while increasing channels, whereas dilated CNNs preserve higher resolution at greater computation cost.The cited examples contrast standard ResNet output at 1/32 input size with dilated ResNet at 1/4.Source: §2.1 · p. 2
  • DeepLab, PSPNet, and EncNet use dilated CNNs, which the paper characterizes as unsuitable for time-efficiency-demanding tasks because of their higher computation cost.This cost is associated with retaining larger spatial feature maps.Source: §2.1 · p. 2

2.2. Real-time Semantic Segmentation

  • Real-time approaches include lightweight model modifications and BiSeNet's shallow spatial and deep context branches; the paper notes accuracy costs in some models and difficulty combining BiSeNet's high-level branch features.ShelfNet is proposed with multiple encoder-decoder pairs and skip connections across spatial levels.Source: §2.2 · pp. 2–3

3. Methods

ShelfNet combines multi-scale backbone features in a shelf of encoder and decoder branches linked by cross-scale skips. Channel reduction and shared-weight residual blocks limit computation and parameters, while numerous information-flow paths provide deep and shallow routes.

3.1. Structure of ShelfNet

  • ShelfNet feeds multi-scale backbone features into a shelf with one downsampling encoder branch, two upsampling decoder branches, and skip connections between adjacent columns at each spatial level.For ResNet, levels A–D correspond to 1/4, 1/8, 1/16, and 1/32 of the input; the S-block combines vertical and lateral inputs while preserving feature-map shape.Source: §3.1 · p. 3

3.2. Channel Reduction for Faster Inference Speed

  • Reducing both convolution channel counts fourfold lowers the stated convolution computation burden to 1/16; with two extra columns, the text gives a relative burden of 1/8.The channel reduction uses 1 × 1 convolution, batch normalization, and ReLU before the segmentation shelf.Source: §3.2 · p. 3

3.3. ShelfNet as an Ensemble of FCNs

  • At four spatial levels, ShelfNet has 29 effective FCN paths versus SegNet's 4, and the path count grows exponentially with encoder-decoder pairs and spatial levels.The figure illustrates example paths, each equivalent to an FCN variant except for backbone pooling layers.Source: §3.3 · pp. 3–4, Figure 2 · p. 3

3.4. Ensemble of Deep and Shallow Paths

  • ShelfNet's deepest path traverses all 16 blocks, compared with 10 blocks for GridNet when the networks stack the same number of encoder-decoder blocks.The paper describes ShelfNet as combining information from shallow and deep paths.Source: §3.4 · p. 4, Figure 3 · p. 4

3.5. Shared-weights Residual Block (S-block)

  • The S-block shares weights between its two convolution layers while retaining separate batch-normalization layers and adding dropout between convolutions.Used in the segmentation shelf, it combines skip connections, recurrent convolution, and dropout regularization with fewer parameters than a standard residual block.Source: §3.5 · p. 4

4. Experiments and Results

The experiments assess ShelfNet's inference speed and segmentation accuracy on three public datasets. They use mIoU for performance and mean running time over 100 single-image repetitions for speed.

4. Experiments and Results

  • Experiments cover PASCAL VOC 2012, PASCAL Context, and Cityscapes to assess inference speed and segmentation accuracy.The study proceeds from dataset and implementation details to ablations and comparisons with state-of-the-art methods.Source: §4 · p. 4
  • Segmentation performance is measured using mean intersection over union (mIoU).The passage identifies mIoU as the performance measure across the experiments.Source: §4 · p. 4
  • Inference speed is measured as the mean running time across 100 repetitions with a single image fed to the network.The passage specifies the repetition count and single-image test procedure.Source: §4 · p. 4

4.1. Datasets and Implementation Details

The experiments use PASCAL VOC, PASCAL-Context, and Cityscapes, with dataset-specific training and augmentation procedures.

4.1.1 PASCAL VOC 2012

  • PASCAL VOC 2012 has 20 object classes and one background class; augmented PASCAL VOC [12] contains 10,582 training, 1,449 validation, and 1,456 test images, with MS COCO [20] used as extra training data for higher accuracy.Source: §4.1.1 · p. 4

4.1.2 PASCAL-Context

  • PASCAL-Context provides 4,998 training images and 5,105 test images with 59 classes plus background.Source: §4.1.2 · p. 4

4.1.3 Cityscapes

  • Cityscapes covers 50 cities and 19 categories, with 2,975 training, 500 validation, and 1,525 fine-labeled test images.More than 20,000 coarsely annotated images are also available.Source: §4.1.3 · p. 4

4.1.4 Implementation Details

  • Training uses SGD and cross-entropy, with dataset-specific schedules; augmentation includes random flips, scaling, rotation, and crops.VOC training proceeds through MS COCO, augmented VOC, then original VOC; PASCAL-Context training runs 80 epochs. Final predictions average multi-scale inputs from 0.5 to 2.Source: §4.1.4 · pp. 4–5

4.2. Ablation Analysis

The ablations associate accuracy with information-flow paths, show shared weights reducing parameters without sacrificing accuracy, and identify dilation as a computation cost.

4.2.1 Ablation Study for Structure

  • Accuracy follows the information-path ordering ShelfNet > W-Net > SegNet > FCN, with W-Net lacking two skip connections used by ShelfNet.Source: §4.2.1 · p. 5, Table 1 · p. 5
  • Halving ShelfNet channels reduces parameters from 38.7M to 29.2M as mIoU falls from 88.26% to 85.52%, still outperforming FCN with 35.0M parameters.Source: §4.2.1 · p. 5

4.2.2 Shared-weights Blocks (S-Block)

  • Shared weights reduce parameters from 45.8M to 38.7M without sacrificing accuracy; mIoU is 88.26% versus 88.02% for conventional residual blocks.Source: §4.2.2 · p. 5, Table 2 · p. 5

4.2.3 Speed Analysis for Backbone

  • About 5× larger computation burden accompanies dilation; ShelfNet’s undilated backbone is identified as the source of its speed advantage.Source: §4.2.3 · pp. 5–6, Table 3 · p. 6

4.3. Results

ShelfNet reports strong accuracy and fast inference across PASCAL VOC, PASCAL-Context, and Cityscapes, including a 79.0% mIoU result on Cityscapes test. Results cover both real-time and non-real-time variants.

4.3.1 PASCAL VOC 2012

  • 4–5× faster inference than PSPNet and EncNet accompanies comparable mIoU; against DeepLabv2, ShelfNet reports 79.3% vs 79.7% mIoU and 103 FPS vs 12 FPS.ShelfNet’s speed advantage over PSPNet and DeepLab is attributed mainly to avoiding dilation in the backbone.Source: §4.3.1 · p. 6
  • 82.8% accuracy from ShelfNet50 exceeds Lightweight-RefineNet with ResNet152 (82.7%) and RefineNet with ResNet101 (82.4%); ShelfNet101 reaches 84.2%.The passage links this performance with using a smaller backbone than the compared models.Source: §4.3.1 · p. 6

4.3.2 PASCAL-Context

  • On PASCAL-Context, without extra training data, ShelfNet reports 45.6% mIoU at 59 FPS and 48.4% mIoU at 42 FPS; RefineNet reports 47.3% mIoU at 29 FPS.Source: §4.3.2 · p. 6

4.3.3 Cityscapes

  • 74.1% mIoU for ShelfNet50 and 77.5% mIoU for ShelfNet101 are reported on Cityscapes; non-real-time results average multi-scale evaluations.The paper distinguishes these results from real-time single-scale evaluation.Source: §4.3.3 · p. 8, Table 8 · p. 8
  • 79.0% mIoU on Cityscapes test is ShelfNet34-lw’s reported result; ShelfNet18-lw reaches 74.8% mIoU and comparable speed to BiSeNet.The real-time evaluation uses single-scale input.Source: §4.3.3 · p. 8, Table 8 · p. 8, Table 9 · p. 8

5. Conclusion

ShelfNet uses multiple encoder-decoder pairs with skip connections between adjacent branches. Across three benchmarks, the paper reports high segmentation accuracy and fast inference, including real-time Cityscapes performance comparable in speed to BiSeNet.

5. Conclusion

  • ShelfNet’s shelf-shaped structure provides multiple paths for information flow, while shared weights in the S-block reduce parameter number without sacrificing accuracy.The paper reports comparable accuracy to state-of-the-art off-line models with 4–5 times faster inference speed.Source: §5 · p. 8

Related papers

Loading 1811.11254v6…