PaperlayerView exact revision on arXiv

Source-linked AI summary

Guided Upsampling Network for Real-Time Semantic Segmentation

Davide Mazzini

arXiv:1807.07466v1cs.CV

TL;DR

Street-scene segmentation must balance processing speed and accuracy, while common upsampling uses parameter-free operators on low-resolution maps. The paper proposes a multiresolution network with Guided Upsampling Module, and reports 70.4% mIoU at 33.3 FPS on Cityscapes test data.

  • Problem

    Street-scene segmentation for automotive use requires fast decisions, while downsampled semantic maps are commonly upsampled with parameter-free operators.

    Source: §1.1 · p. 1, §3 · p. 3, §4 · p. 6

  • Method

    Guided Upsampling Network combines multiresolution features with GUM, which guides semantic-map upsampling using high-resolution details.

    Source: §1.1 · p. 2, §4 · p. 6

  • Results

    70.4% mIoU at 33.3 FPS on Cityscapes test data on a single Titan Xp GPU.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The network processes high-resolution street-scene images in real time while attaining the reported Cityscapes test performance.

    Source: §7 · p. 9

    The module for exploiting high-resolution information is computationally demanding because of the number of required layers.

    Source: §4 · p. 8

Abstract

from arXiv · show

Semantic segmentation architectures are mainly built upon an encoder-decoder structure. These models perform subsequent downsampling operations in the encoder. Since operations on high-resolution activation maps are computationally expensive, usually the decoder produces output segmentation maps by upsampling with parameters-free operators like bilinear or nearest-neighbor. We propose a Neural Network named Guided Upsampling Network which consists of a multiresolution architecture that jointly exploits high-resolution and large context information. Then we introduce a new module named Guided Upsampling Module (GUM) that enriches upsampling operators by introducing a learnable transformation for semantic maps. It can be plugged into any existing encoder-decoder architecture with little modifications and low additional computation cost. We show with quantitative and qualitative experiments how our network benefits from the use of GUM module. A comprehensive set of experiments on the publicly available Cityscapes dataset demonstrates that Guided Upsampling Network can efficiently process high-resolution images in real-time while attaining state-of-the art performances.

1 Introduction

Encoder–decoder segmentation models balance context and computation by downsampling, then commonly upsample low-resolution semantic maps with parameter-free operators. This work proposes a multiresolution network and a guided upsampling module for street-scene segmentation, alongside a review of related accuracy- and efficiency-oriented designs.

1 Introduction

  • Segmentation models downsample to enlarge receptive fields and reduce computation, then commonly upsample 1/8- or 1/16-resolution maps with nearest-neighbor or bilinear interpolation.Even architectures using dilated convolutions retain downsampling to keep computation feasible.Source: §1 · p. 1

1.1 Our focus and contribution

  • The design targets automotive street-scene segmentation, where models must run continuously and quickly respond to environmental events.The authors frame their choices as a trade-off between processing speed and accuracy.Source: §1.1 · pp. 1–2
  • The proposed network reaches 70.4% IoU on Cityscapes test images at 33 FPS on one GPU, combining high-quality predictions with speed.The contribution describes a 512x1024 input image and a single GPU for the 33 FPS result.Source: §1.1 · p. 2
  • Guided Upsampling Module (GUM) uses high-resolution clues during upsampling, while the network jointly exploits high-resolution textures and large-context information.The architecture uses a multiresolution design, and GUM learns a transformation conditioned on high-resolution details.Source: §1.1 · p. 2, Figure 1 · p. 2

1.2 Related works

  • Related work spans high-accuracy approaches using skip connections, dilated convolutions, deep encoders, or context layers, often at high computational cost.FCN [15] uses skip connections; DeepLab [2] uses dilated convolutions; PSPNet [25] and DeepLabv3 [5] add context layers.Source: §1.2 · p. 2
  • Efficiency-oriented designs pursue speed through approaches including ENet [16], SegNet, ICNet [24], and ERFNet, with ENet sacrificing accuracy.SegNet reuses max-pooling indices during upsampling; ICNet [24] uses three branches and deep supervision.Source: §1.2 · p. 3

2 Dataset and evaluation metrics

The experiments use Cityscapes urban-scene images and report segmentation accuracy and inference speed.

2 Dataset and evaluation metrics

  • Cityscapes has 5,000 finely annotated 2048x1024 images; evaluation uses 19 classes, mIoU, and single-Titan-Xp FPS.The train, validation, and test sets contain 2,975, 500, and 1,525 images, respectively; FPS is the inverse of one forward-pass time.Source: §2 · p. 3

3 Network design

The network balances speed and accuracy with a multiresolution encoder and a lightweight decoder, then tests design choices incrementally. Its experiments favor shared encoder branches and fusion post-processing, while training augmentation retains random scaling.

3 Network design

  • The encoder combines a low-resolution branch for large-context features with a medium-resolution branch for local features that help recover detail during decoding.The branches process inputs at different subsampling factors; the shared enc24 configuration was selected as the base encoder.Source: §3 · pp. 3–4
  • Input subsampling raises speed from 6.7 FPS to 50.6 FPS but causes an 8% performance drop.The comparison uses the baseline model with bilinear upsampling and the same model trained and evaluated on inputs subsampled by factor 4.Source: §3 · p. 3
  • Shared encoder branches perform better than unshared branches, while adding a full-resolution branch improves performance but slows the system below 30 FPS.The authors therefore use the shared two-branch configuration as the base encoder and discard the three-branch configuration.Source: §3 · p. 4
  • Fusion post-processing improves both mIoU and speed by adding convolutions and nonlinearities before upsampling from a smaller feature space.The tested fusion designs vary signal summation versus concatenation and whether post-processing is included.Source: Table 2 · p. 4, §3 · p. 5

3.1 Training recipes

  • Training uses SGD with momentum and batch size 8; among the tested augmentations, only random scaling improves results and is retained.The learning rate starts at 0.001 and decreases by an order of magnitude every 100 epochs; the authors report that increasing batch size hurts performance in their setup.Source: §3.1 · pp. 5–6, Table 3 · p. 5

4 Guided upsampling module

GUM guides upsampling with learned offsets that steer sampling toward the correct semantic class. The Guidance Module predicts these offsets, while three designs trade off high-resolution information, receptive field, and computation.

4 Guided upsampling module

  • GUM uses a Guidance Module to predict a high-resolution offset table that steers nearest-neighbor sampling toward the correct semantic class.The module can also be adapted to bilinear upsampling.Source: §4 · p. 6
  • GUM is interpreted as a Spatial Transformer Network in which the Guidance Module combines the localization and grid-generation roles, while grid sampling retains the same role.The Guidance Module is a trainable branch that predicts the offset table end-to-end by backpropagation.Source: §4 · p. 7
  • The large-rf Guidance Module relies on deep activations without high-resolution information and is the most computationally demanding design.Its computation reflects the number of layers required.Source: §4 · pp. 7–8
  • The high-res Guidance Module maps a 32-dimensional feature space to two offset dimensions with a 1x1 convolution, adding 64 parameters and the same number of per-pixel operations.In this architecture, the authors characterize its computational cost as almost free.Source: §4 · p. 8
  • Fusion Guidance Module achieves the best performance among the tested designs while requiring only two Conv-BatchNorm blocks and one upsampling layer.It combines high-resolution and large-receptive-field activation maps and is faster than iterative upsampling in the large-rf design.Source: §4 · p. 8, Table 4 · p. 8

5 Boundaries analysis

The Cityscapes trimap experiment evaluates predictions at increasing distances from object boundaries. GUM improves on bilinear upsampling, with fusion achieving slightly higher mIoU than the other Guidance Module designs.

5 Boundaries analysis

  • GUM shows a clear advantage over the same network using bilinear upsampling in the Cityscapes trimap experiment.Across the tested Guidance Modules, fusion achieves slightly higher mIoU levels.Source: §5 · p. 9
  • The experiment evaluates pixels at progressively greater distances from object boundaries using thresholds on ground-truth distance-transform maps.The authors describe this as a different but equivalent implementation of the trimap experiment because Cityscapes lacks a boundary class to expand.Source: §5 · p. 9
  • The authors report this as the first trimap experiment on Cityscapes, adapting the approach used on Pascal VOC.The Pascal VOC version expanded annotation borders and evaluated pixels in the expanded boundary region.Source: §5 · pp. 8–9

6 Comparison with the state-of-the-art

On Cityscapes test, Guided Upsampling Network reaches 70.4% mIoU without postprocessing and, to the authors’ knowledge, has the highest mIoU among published methods running above 30 FPS. The comparison distinguishes this speed-oriented result from accuracy-focused methods that use time-consuming multi-scale testing.

6 Comparison with the state-of-the-art

  • 70.4% mIoU without postprocessing is, to the authors’ knowledge, the highest result among published methods running at >30 FPS.The evaluation is on the Cityscapes test set.Source: §6 · p. 9
  • 81.2% mIoU makes DeepLabv3+ the best published accuracy-oriented model cited, but it uses time-consuming multi-scale testing.Source: §6 · p. 9
  • Guided Upsampling Network performs better than Adelaide and Dilation10, which the authors describe as methods that do not prioritize speed.Source: §6 · p. 9

7 Conclusions

The paper concludes that its multiresolution Guided Upsampling Network combines high-resolution textures with large-context information for real-time street-scene segmentation. It reports 70.4% mIoU at 33.3 FPS on a single Titan Xp GPU, and introduces GUM to condition upsampling on high-resolution details.

7 Conclusions

  • 70.4% mIoU at 33.3 FPS on a single Titan Xp GPU is the reported Cityscapes test result.Source: §7 · p. 9
  • The multiresolution architecture jointly exploits high-resolution textures and large-context information for real-time street-scene segmentation.Source: §7 · p. 9
  • GUM improves upsampling by learning a transformation conditioned on high-resolution details, with low additional computational cost.Source: §7 · p. 9
  • Table 5 compares methods on the Cityscapes test set, sorted by increasing mIoU, and marks the proposed method in boldface.Source: Table 5 · p. 10

Related papers

Loading 1807.07466v1…