PaperlayerView exact revision on arXiv

Source-linked AI summary

SPGNet: Semantic Prediction Guidance for Scene Parsing

Bowen Cheng, Liang-Chieh Chen, Yunchao Wei, Yukun Zhu, Zilong Huang, Jinjun Xiong, Thomas Huang, Wen-Mei Hwu, Honghui Shi

arXiv:1908.09798v1cs.CV

TL;DR

Multi-stage encoder-decoder networks have been used in pose estimation but rarely explored for semantic segmentation. SPGNet uses pixel-wise semantic predictions to guide feature re-weighting across stages; on Cityscapes, it outperforms DenseASPP using fine annotations and achieves a better accuracy/speed trade-off than DANet, requiring 22.7% of its computation.

  • Problem

    Multi-stage encoder-decoder networks are rarely explored for semantic segmentation, despite their use in pose estimation and the reported advantage over single-stage counterparts.

    Source: §1 · p. 2

  • Method

    SPG re-weights local features using guidance from pixel-wise semantic predictions in a multi-stage encoder-decoder network.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    SPGNet outperforms DenseASPP on the Cityscapes test set using fine annotations and requires 22.7% of DANet’s computation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A two-stage encoder-decoder network with SPG significantly outperforms its one-stage counterpart with similar parameters and computation.

    Source: §5 · p. 8

    SPGNet requires supervision to learn Guided Attention.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

Multi-scale context module and single-stage encoder-decoder structure are commonly employed for semantic segmentation. The multi-scale context module refers to the operations to aggregate feature responses from a large spatial extent, while the single-stage encoder-decoder structure encodes the high-level semantic information in the encoder path and recovers the boundary information in the decoder path. In contrast, multi-stage encoder-decoder networks have been widely used in human pose estimation and show superior performance than their single-stage counterpart. However, few efforts have been attempted to bring this effective design to semantic segmentation. In this work, we propose a Semantic Prediction Guidance (SPG) module which learns to re-weight the local features through the guidance from pixel-wise semantic prediction. We find that by carefully re-weighting features across stages, a two-stage encoder-decoder network coupled with our proposed SPG module can significantly outperform its one-stage counterpart with similar parameters and computations. Finally, we report experimental results on the semantic segmentation benchmark Cityscapes, in which our SPGNet attains 81.1% on the test set using only 'fine' annotations.

1. Introduction

The paper introduces SPG to re-weight local features using pixel-wise semantic predictions, and explores multi-stage encoder-decoder networks for semantic segmentation. On Cityscapes, SPGNet outperforms DenseASPP with fine annotations and uses 22.7% of DANet’s computation.

1. Introduction

  • Semantic segmentation assigns a semantic label to every image pixel, and deep convolutional networks have improved its performance.The introduction describes multi-scale context modules as aggregating feature responses over large spatial extents to enrich local features.Source: §1 · p. 1
  • SPG re-weights local feature values using pixel-wise semantic predictions in a supervise-and-excite framework.Semantic supervision is injected into feature maps, then a 1 × 1 convolution and sigmoid produce Guided Attention to re-weight transformed features; an identity mapping is also added.Source: §1 · pp. 1–2
  • A two-stage encoder-decoder network with SPG significantly outperforms its one-stage counterpart at similar parameters and computation.The paper revisits multi-stage encoder-decoder networks for Cityscapes semantic segmentation.Source: §1 · p. 2
  • 22.7% of DANet’s computation is required by SPGNet, whose class-wise mIoU exceeds DANet’s in 14 of 19 classes despite slightly lower overall mIoU.On the Cityscapes test set using only fine annotations, SPGNet also outperforms DenseASPP.Source: §1 · p. 2
  • The paper explores multi-stage encoder-decoder networks for semantic segmentation and reports that adding SPG further improves their performance.It also provides ablation studies and visualizations of learned attention maps.Source: §1 · p. 2

2. Related Works

Related work contrasts SPG’s pixel-level semantic guidance with context-based attention, and situates the multi-stage design among methods for pose estimation and segmentation. The cited segmentation approaches include systems that pass features across stages without passing their predictions.

2. Related Works

  • Multi-stage networks refine estimates iteratively in human pose estimation; for semantic segmentation, Stacked Deconvolutional Network passes features across stages but neglects stage predictions.The paper proposes using predictions as a special attention to propagate useful features across stages.Source: §2 · pp. 2–3
  • Semantic segmentation methods use encoder-decoder structures to capture context in the encoder and recover high-resolution features in the decoder, alongside context modules such as pyramid pooling and atrous convolutions.The related-work discussion also describes atrous convolution as computationally expensive for extracting output feature maps at one-eighth or one-quarter input resolution.Source: §2 · p. 2
  • SPG uses pixel-level semantic supervision to guide local features, unlike SENets’ channel re-weighting from spatially aggregated features, GENets’ long-range spatial gathering, and EncNet’s image-level supervision.The paper also distinguishes SPG from self-attention and other attention mechanisms for merging or selecting features.Source: §2 · p. 3

3. Methods

SPGNet stacks encoder-decoder stages, passing earlier-stage outputs through SPG and aggregating cross-stage features. Its decoder combines encoder features with upsampled decoder outputs, while SPG turns semantic predictions into guidance for the next stage.

3.1. Overall Architecture

  • Each SPGNet stage has an encoder that produces multi-scale dense features and a GAP image-level vector, followed by a decoder that recovers spatial resolution.Earlier decoder output feeds SPG for the next stage, and cross-stage feature aggregation uses earlier encoder and decoder features; the final decoder output is bilinearly upsampled to image resolution for per-pixel predictions.Source: §3.1 · p. 3

3.2. Encoder / Decoder Design

  • The encoder-decoder design follows FPN principles, using 256-channel decoder features, GAP-derived image-level features, and an efficient upsample module.The paper contrasts this design with symmetric hourglass networks, whose equal encoder-decoder computation prevents use of ImageNet-pretrained weights.Source: §3.1 · p. 3, §3.2 · p. 4

3.3. Upsample module

  • Each upsample module transforms an encoder feature with a residual block, adds it to a bilinearly upsampled decoder feature, then uses another residual block to fuse them.The module combines features from the encoder and previous decoder layer.Source: §3.3 · p. 4

3.4. Semantic Prediction Guidance

  • SPG converts previous-stage decoder features into per-class logits, then into a sigmoid Guided Attention mask that re-weights transformed features before producing the next-stage encoder input.Training minimizes the final-stage prediction loss and the per-class-logit losses from all previous stages; SPG adds about 1% FLOPs and 2.3% peak memory usage while making re-weighting more interpretable.Source: §3.4 · p. 4

4. Experiments

On Cityscapes, SPGNet offers a strong accuracy-computation trade-off, while ablations identify effective design choices for its two-stage model. Visualizations and tests on other datasets show both the reach and limits of guided attention.

4.1. Dataset

  • On Cityscapes, experiments use 5,000 fine-annotated images split into 2,975 training, 500 validation, and 1,525 test images.The paper uses fine annotations only.Source: §4.1 · p. 4

4.2. Implementation Details

  • The best Cityscapes model uses two encoder-decoder stages with ResNet-50 and 256 decoder channels; training uses SGD and augmentation, with 80,000 iterations for ablations.For test evaluation, the model is trained on the concatenated train and validation sets.Source: §4.2 · pp. 4–5, §4.3 · p. 5

4.3. Comparison with State-of-the-Arts

  • SPGNet uses 22.7% of DANet’s computation with 0.4 mIoU degradation, and is 0.5% mIoU better than DenseASPP while using almost half its computation.The comparison uses Cityscapes test results with fine annotations and multi-scale inference.Source: §4.3 · p. 5, Table 1 · p. 5
  • SPGNet outperforms DenseASPP in 15 of 19 Cityscapes classes and DANet in 14 of 19, but trails both on Truck IoU.Truck IoU is 10.7 points below DenseASPP and 9.5 below DANet.Source: §4.3 · p. 5, Table 2 · p. 6

4.4. Ablation Studies

  • On Cityscapes validation, sigmoid SPG reaches 77.67% mIoU, 1.36 points above baseline; removing identity mapping or supervision lowers performance.The passage also reports that supervised SPG outperforms the unsupervised SE and GE alternatives.Source: §4.4 · p. 5, Table 3 · p. 6, Table 4 · p. 6
  • Two stages are optimal in the reported ablation; a stronger first-stage encoder outperforms the reverse ResNet-18/ResNet-50 ordering at similar computation and parameter counts.The authors report that deeper encoders generally improve performance and that SPG benefits from OHEM.Source: §4.4 · p. 6, Table 6 · p. 6, Table 7 · p. 6

4.5. Visualization of Guided Attention

  • Guided attention finds many thin Poles and distinguishes Person from Rider, but often activates Truck, Bus, and Train together.The authors connect this confusion to weaker discrimination and poorer performance on those classes.Source: §4.5 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8

4.6. Generalization to Other Datasets

  • On PASCAL VOC 2012, SPGNet reaches 77.33 mIoU, comparable to ResNet-101 DeepLabV3+ at 77.37; on Person-Part it reaches 67.23 single-scale and 68.36 multi-scale.The Person-Part model does not require the extra MPII training data used by some compared approaches.Source: §4.6 · pp. 7–8

5. Conclusion

SPGNet uses semantic prediction to reweight local features and demonstrates state-of-the-art semantic segmentation performance on Cityscapes. Paired with the SPG module, a two-stage encoder-decoder significantly outperforms its one-stage counterpart with similar parameters and computations.

5. Conclusion

  • SPGNet demonstrates state-of-the-art semantic segmentation performance on Cityscapes.Source: §5 · p. 8
  • SPG uses a “supervise-and-excite” framework to reweight local features through semantic prediction, and its Guided Attention maps visualize that reweighting.Source: §5 · p. 8
  • A two-stage encoder-decoder paired with SPG significantly outperforms its one-stage counterpart with similar parameters and computations.Source: §5 · p. 8

A. Extra Ablation Studies

The extra ablations compare decoder upsampling designs and inference-time pooling strategies, reporting efficiency and performance trade-offs.

A. Extra Ablation Studies

  • The proposed upsample module is almost 1.5 mIoU better than the FPN-style module and requires 10B fewer FLOPs, with similar parameters.The comparison uses a ResNet-18 encoder, no global average pooling, and 128 decoder channels.Source: A. Extra Ablation Studies · p. 9
  • AP is more efficient than TILED and performs slightly better because it processes no overlapped regions.TILED processes redundant pixels in overlapping patches and requires heuristics to merge their predictions.Source: A. Extra Ablation Studies · p. 9
  • GAP improves performance only slightly, by 1.3%, under the asymmetric setting of 769 × 769 training crops and 1024 × 2048 inference images.Source: A. Extra Ablation Studies · p. 9

Related papers

Loading 1908.09798v1…