PaperlayerView exact revision on arXiv

Source-linked AI summary

Cars Can't Fly up in the Sky: Improving Urban-Scene Segmentation via Height-driven Attention Networks

Sungha Choi, Joanne T. Kim, Jaegul Choo

arXiv:2003.05128v3cs.CV

TL;DR

Urban-scene segmentation models generally do not encode the strong class-distribution differences associated with vertical image position. The paper proposes HANet, a lightweight add-on that uses height-wise context to scale features, and reports consistent gains across backbones and datasets, including a new Cityscapes state of the art among ResNet-101-based models.

  • Problem

    Urban-scene images have vertical positional class patterns that common semantic-segmentation architectures do not explicitly exploit.

  • Method

    HANet extracts height-wise contextual information and predicts position-dependent channel scaling for features or classes within horizontal image regions.

  • Results

    HANet consistently improves baseline models across various backbones and datasets, while achieving new state-of-the-art Cityscapes performance among ResNet-101-based models.

  • Takeaways & Limitations

    HANet provides a broadly applicable, cost-effective way to exploit urban-scene spatial priors, with negligible computational and memory overhead.

Abstract

from arXiv · show

This paper exploits the intrinsic features of urban-scene images and proposes a general add-on module, called height-driven attention networks (HANet), for improving semantic segmentation for urban-scene images. It emphasizes informative features or classes selectively according to the vertical position of a pixel. The pixel-wise class distributions are significantly different from each other among horizontally segmented sections in the urban-scene images. Likewise, urban-scene images have their own distinct characteristics, but most semantic segmentation networks do not reflect such unique attributes in the architecture. The proposed network architecture incorporates the capability exploiting the attributes to handle the urban scene dataset effectively. We validate the consistent performance (mIoU) increase of various semantic segmentation models on two datasets when HANet is adopted. This extensive quantitative analysis demonstrates that adding our module to existing models is easy and cost-effective. Our method achieves a new state-of-the-art performance on the Cityscapes benchmark with a large margin among ResNet-101 based segmentation models. Also, we show that the proposed model is coherent with the facts observed in the urban scene by visualizing and interpreting the attention map. Our code and trained models are publicly available at https://github.com/shachoi/HANet

1. Introduction

Urban-scene images exhibit strong vertical class-distribution patterns that standard segmentation architectures do not explicitly exploit. HANet uses this positional structure to improve segmentation through lightweight, broadly applicable channel scaling.

  • Motivation: Urban-scene images contain positional patterns because front-mounted cameras produce road-driving views with spatially structured visual content.These properties motivate incorporating vertical spatial priors into segmentation architectures.
  • Motivation: 88% of the dataset is dominated by five classes, whose probabilities change sharply across upper, middle, and lower image regions.The dominant classes are road, building, vegetation, car, and sidewalk.
  • Motivation: 36.9% is the road-class probability for an entire image, versus 0.006% in the upper region and 87.9% in the lower region.These values show that vertical position substantially changes class likelihoods.
  • Motivation: 1.84 versus 1.26: conditional entropy decreases when pixels are conditioned on three horizontal regions rather than the entire image.The lower uncertainty supports identifying a pixel’s horizontal region for pixel-level classification.
  • Proposed approach: HANet extracts height-wise contextual information and scales feature channels according to vertical position, predicting which features or classes matter within each horizontal part.It is designed as a general add-on module for semantic segmentation of urban-scene images.
  • Results: HANet consistently improves baseline models across backbones and achieves state-of-the-art Cityscapes performance among ResNet-101-based models with negligible computational and memory overhead.The paper also visualizes attention weights to support the role of height position in urban-scene segmentation.

2. Related Work

Prior semantic-segmentation work addresses resolution, receptive field, long-range dependencies, boundaries, context, object scale, and urban-scene spatial properties through specialized architectural or adaptation techniques.

  • Model architectures: Deconvolution and skip-connections recover resolution or object boundaries, while atrous convolution, ASPP, and pyramid pooling expand receptive fields or handle scale variation.These approaches modify feature resolution, context range, or multiscale processing.
  • Context and boundaries: Self-attention methods capture long-range dependencies, and separate boundary modules target boundary processing or adaptive downsampling.These methods focus on relationships across distant features or on boundary quality.
  • Urban-scene exploitation: Urban-scene methods address heterogeneous object scales, spatial priors, region-specific domain adaptation, and correlations between depth and semantic information.These approaches explicitly exploit properties of urban-scene imagery or its spatial organization.
  • Channel-wise attention: Channel-wise attention methods model inter-channel relationships and scale feature maps using channel importance, including global-context mechanisms such as SENets.HANet is positioned as closely connected to this family of attention approaches.

3. Proposed Method

HANet extracts height-wise context from urban-scene feature maps and uses it to generate row-dependent channel attention. The module pools, resamples, processes, and injects positional information before gating higher-level representations.

  • HANet generates per-channel scaling factors for each image row from height-wise contextual information.
  • The module derives a channel-wise attention map from lower-level features and fits it to the channel and height dimensions of higher-level features.Its pipeline includes width-wise pooling, interpolation for coarse attention, attention-map computation, and optional positional encoding.
  • Width-wise pooling aggregates each row of the input feature map into a one-column representation, with average pooling selected empirically over max pooling.
  • The pooled representation is downsampled to a coarse height and later upsampled to match the higher-level feature map.The coarse height ˆH is empirically set to 16.
  • Convolutional layers compute attention values for each row, and sigmoid activation allows multiple informative features or labels to be weighted simultaneously.This differs from softmax-based single-choice normalization because a row may correspond to multiple labels such as road, car, and sidewalk.
  • Sinusoidal positional encodings are added to intermediate feature maps, while height positions may be randomly jittered by up to two positions for cross-dataset camera variation.

4. Experiments

Experiments evaluate HANet across backbones, inference settings, image regions, datasets, and attention visualizations. Results show consistent accuracy gains with negligible added complexity, including strong Cityscapes and BDD100K performance.

  • Experimental setup: HANet is evaluated on Cityscapes and BDD100K using DeepLabv3+ with ShuffleNetV2, MobileNetV2, and ResNet backbones.Performance is measured with mean Intersection over Union (mIoU).
  • Effectiveness of HANet: Adding HANet consistently improves mIoU across backbone networks while adding practically negligible parameter and FLOP costs.The comparison indicates that HANet offers a more favorable accuracy-complexity trade-off than changing output stride.
  • Ablation studies: 79.31 mIoU is achieved after adding HANet with positional encodings at multiple layers and reduction ratio 32, compared with 77.80 without that configuration.The ablation uses ResNet-101 with output stride 16 on the Cityscapes validation set.
  • Inference techniques: 82.05% mIoU is achieved by the best model on the Cityscapes validation set using flipped, multiscale, and sliding inference.The multiscale scales are {0.5, 1.0, 2.0}.
  • Efficacy at segmented regions: HANet yields larger performance gains in the upper and lower image regions, where horizontal subdivision reduces entropy more than in the middle region.The regional result is reported for four horizontal sections on the Cityscapes validation set.
  • Benchmark comparisons: The models achieve new state-of-the-art performance on the Cityscapes test set, while HANet also produces superior results on BDD100K validation data.The Cityscapes comparison includes ResNet-101 and ResNeXt-101 models; BDD100K includes varied driving conditions.
  • Qualitative analysis: Attention visualizations show that HANet assigns different weights by vertical position, following height-wise class distributions such as road in lower regions and sky in upper rows.These visualizations support that the model captures structural priors and intrinsic urban-scene features.

5. Conclusions

The paper presents HANet as an effective, broadly applicable add-on for urban-scene segmentation. It reports performance gains with negligible cost and uses attention-map analysis to support the value of vertical positional information.

  • HANet is presented as an effective and widely applicable add-on module for urban-scene segmentation.
  • Adding HANet to baseline models increases performance with negligible computational cost.
  • Attention-map visualization and analysis support the hypothesis that vertical positional information helps urban-scene semantic segmentation.

A. Supplementary Material

The supplementary material adds comparisons, positional-information experiments, architecture details, and analyses of height-wise and width-wise class distributions.

  • The supplement first provides additional comparisons with other state-of-the-art models.
  • It then evaluates ways to incorporate positional information and describes the baseline and HANet architectures.
  • The supplement compares height-wise and width-wise class distributions and includes further quantitative results and visualizations.

A.1. Additional comparisons with other models

The supplementary comparison evaluates the best HANet performance against published state-of-the-art results on the Cityscapes validation set, focusing on ResNet-101 models.

  • The comparison uses the Cityscapes validation set.
  • It compares the best performance of HANet with performances reported for other state-of-the-art models.
  • The compared models are based on ResNet-101.

A.2. Positional encoding and embedding.

The supplementary experiment studies how positional encoding is incorporated into HANet. It compares layers and encoding methods on Cityscapes using ResNet-101 with output stride 8.

  • Sinusoidal positional encoding in HANet's second convolutional layer gives the best result among the tested options.
  • The experiment compares positional-encoding layers and methods in HANet.HANet consists of three convolutional layers.
  • The evaluation uses ResNet-101 with output stride 8 on the Cityscapes validation set.

A.3. Further implementation details

The implementation details describe HANet, the baseline architecture, and the evaluation setup used for per-class IoU and mIoU comparison.

  • HANet implementation: HANet uses three convolutional layers with dropout and batch normalization, extracting height-wise context through average pooling.The module is described as an add-on implemented alongside the baseline segmentation model.
  • HANet implementation: Width-wise pooling and interpolation produce coarse attention before dropout and three one-dimensional convolutional layers refine it.The coarse-attention height and channel-reduction ratio are set as hyperparameters.
  • Baseline architecture: The baseline follows DeepLabv3+ with encoder-decoder skip connections and an auxiliary branch after ResNet stage 3.The auxiliary loss is included to facilitate learning in deep networks.
  • Evaluation setup: Table 10 compares per-class IoU and mIoU for ResNet-101 with output stride 8 on the Cityscapes validation set.Sliding, multi-scale, and flipping inference techniques are applied.

A.4. Height- and width-wise class distribution

Urban-scene class distributions vary distinctly by vertical position but remain relatively similar across horizontal positions, motivating height-wise rather than width-wise attention.

  • Height- and width-wise class distribution: Height-wise class distributions show distinct patterns across vertical positions, unlike width-wise distributions.The distributions represent normalized probabilities for the 19 classes.
  • Height- and width-wise class distribution: Width-wise attention produced no meaningful performance increase empirically.The relative similarity across columns makes distinct horizontal-position information difficult to extract.
  • Height- and width-wise class distribution: HANet therefore extracts and incorporates height-wise contextual information instead of width-wise context.This design choice follows the observed difference between vertical and horizontal class distributions.

A.5. Per-class IoU and segmentation maps

The evaluation examines per-class IoU and segmentation maps, finding broad class-level improvements and clearer predictions for visually confusing or fragmented objects.

  • Per-class IoU: Up to 7.3% IoU increase is observed, while all classes show similar or improved results versus the baseline.The comparison uses per-class IoU and mIoU results.
  • Segmentation maps: HANet more accurately distinguishes nearby vehicles, vegetation from terrain, and poles from buildings in qualitative comparisons.The cited comparisons include trucks, buses, cars, vegetation, terrain, poles, and buildings.
  • Segmentation maps: The method generally forms clearer object boundaries while avoiding unnecessary fragmentation into multiple pieces.The authors conjecture that height-wise contextual gating helps separate confusing classes according to vertical position.
  • Segmentation maps: The figures compare predicted segmentation maps across confusing pairs including vehicles, vegetation and terrain, fence and vegetation, and sky and building.These examples provide qualitative cases for assessing class separation.
Loading 2003.05128v3…