PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes

Yuanduo Hong, Huihui Pan, Weichao Sun, Yisong Jia

arXiv:2101.06085v2cs.CV

TL;DR

Real-time road-scene segmentation must close the accuracy gap between lightweight methods and accurate but costly dilation backbones. The paper proposes DDRNets with dual-resolution branches and DAPPM, achieving strong accuracy–speed trade-offs on Cityscapes and CamVid, including 77.4% mIoU at 102 FPS on Cityscapes.

  • Problem

    Real-time segmentation methods remain less accurate than dilation-backbone models, while high-resolution scene parsing is computationally expensive for autonomous driving.

  • Method

    DDRNets use deep high- and low-resolution branches with multiple bilateral fusions, while DAPPM aggregates multi-scale context from low-resolution feature maps.

  • Results

    DDRNets achieve a new accuracy–speed trade-off on Cityscapes and CamVid, with DDRNet-23-slim reaching 77.4% mIoU at 102 FPS on Cityscapes test set.

  • Takeaways & Limitations

    The approach provides scalable real-time backbones that outperform previous real-time models across three benchmarks without attention mechanisms or extra bells and whistles.

Abstract

from arXiv · show

Semantic segmentation is a key technology for autonomous vehicles to understand the surrounding scenes. The appealing performances of contemporary models usually come at the expense of heavy computations and lengthy inference time, which is intolerable for self-driving. Using light-weight architectures (encoder-decoder or two-pathway) or reasoning on low-resolution images, recent methods realize very fast scene parsing, even running at more than 100 FPS on a single 1080Ti GPU. However, there is still a significant gap in performance between these real-time methods and the models based on dilation backbones. To tackle this problem, we proposed a family of efficient backbones specially designed for real-time semantic segmentation. The proposed deep dual-resolution networks (DDRNets) are composed of two deep branches between which multiple bilateral fusions are performed. Additionally, we design a new contextual information extractor named Deep Aggregation Pyramid Pooling Module (DAPPM) to enlarge effective receptive fields and fuse multi-scale context based on low-resolution feature maps. Our method achieves a new state-of-the-art trade-off between accuracy and speed on both Cityscapes and CamVid dataset. In particular, on a single 2080Ti GPU, DDRNet-23-slim yields 77.4% mIoU at 102 FPS on Cityscapes test set and 74.7% mIoU at 230 FPS on CamVid test set. With widely used test augmentation, our method is superior to most state-of-the-art models and requires much less computation. Codes and trained models are available online.

I. INTRODUCTION

Semantic segmentation for autonomous driving must combine high-resolution spatial detail and large receptive fields, but conventional accurate models are too computationally expensive for real-time deployment. DDRNets address this gap with dual-resolution backbones and DAPPM, achieving strong speed–accuracy trade-offs.

  • Motivation: High-resolution feature maps and large receptive fields are needed for satisfactory dense predictions but make autonomous-driving inference computationally expensive.Large input images and multi-scale testing further increase inference time, limiting direct deployment in vehicles.
  • Motivation: Recent real-time methods use lightweight architectures, including encoder-decoder and bilateral designs, but their quality and scalability remain limited.BiSeNetV2 reaches 72.6% test mIoU at 156 FPS on Cityscapes, while some approaches rely on extra data or deliberately tuned architectures.
  • Proposed approach: DDRNets use two deep branches at different resolutions with multiple bilateral fusions to preserve high-resolution detail while extracting semantic information efficiently.The architecture is designed for high-resolution road-scene images and can be pretrained on ImageNet before segmentation training.
  • Proposed approach: DAPPM combines feature aggregation with pyramid pooling on low-resolution feature maps to extract multi-scale context with little additional inference time.The module merges contextual features in a cascaded way while enlarging effective receptive fields.
  • Results: 77.4% mIoU at 102 FPS on Cityscapes and 74.7% mIoU at 230 FPS on CamVid demonstrate the reported speed–accuracy trade-off.The results are measured on a single 2080Ti GPU and are reported as state-of-the-art trade-offs in the paper.

II. RELATED WORK

Prior semantic-segmentation research includes high-performance dilation backbones, lightweight encoder-decoder and two-pathway methods, and dedicated context modules. DDRNets extend this landscape with compact dual-resolution representations that retain high-resolution features while extracting contextual information.

  • Related-work categories: Dilation-based backbones improve semantic segmentation in challenging scenes, while lightweight encoder-decoder and two-pathway architectures target efficiency.Contextual information is also treated as important for scene parsing.
  • High-performance semantic segmentation: Dilation backbones preserve spatial detail and receptive fields but increase high-resolution feature-map dimensions and create real-time inference challenges.The paper characterizes many state-of-the-art dilation-based models as poorly suited to self-driving scene parsing.
  • High-performance semantic segmentation: Decoder-based substitutes such as DeepLabv3plus fuse upsampled features with low-level features, reducing the need for directly generated high-resolution feature maps.The passage contrasts this strategy with dilation-based backbones.
  • DDRNets: DDRNets maintain deep thin high-resolution representations while simultaneously extracting high-level contextual information through two compact trunks.This design is presented as a more compact alternative within the paper’s real-time segmentation setting.
  • Real-time semantic segmentation: Real-time segmentation commonly relies on encoder-decoder or two-pathway methods, with lightweight encoders playing a central role in both.DDRNets differ from existing two-pathway methods by enabling multiple feature fusions and sufficient ImageNet pretraining while preserving efficiency.

1) Encoder-decoder Architecture:

Encoder-decoder networks reduce computation by processing context at low resolution and restoring spatial detail, but repeated downsampling can impair segmentation accuracy. DDRNets address this trade-off with a deep high-resolution branch, multiple fusions, and scalable design.

  • Encoder-decoder Architecture: Encoder-decoder architectures reduce computation by repeatedly lowering spatial resolution and restoring it through interpolation or transposed convolution.Lightweight ImageNet-pretrained or efficient-from-scratch encoders can further improve efficiency.
  • Encoder-decoder Architecture: Repeated downsampling loses information that upsampling cannot completely recover, impairing semantic segmentation accuracy.
  • Encoder-decoder Architecture: DDRNets use a deep, thin high-resolution branch alongside a semantic branch to enable multiple feature fusions and sufficient ImageNet pre-training.The design aims to retain inference efficiency while improving over existing two-pathway methods.
  • Encoder-decoder Architecture: Lightweight backbones such as MobileNet and ShuffleNet reduce computation, but depthwise separable convolutions may run inefficiently on existing GPU architectures.Their classification-oriented design may also be suboptimal for semantic segmentation.

C. Context Extraction Modules

The method combines a deep dual-resolution network with contextual processing on low-resolution features. Its branches exchange information through bilateral fusion while the architecture is organized for efficient segmentation inference.

  • Context Extraction Modules: The pipeline consists of a Deep Dual-resolution Network and a Deep Aggregation Pyramid Pooling Module.
  • Context Extraction Modules: A high-resolution branch preserves spatial detail while a low-resolution branch extracts semantic information through repeated downsampling.The high-resolution branch can operate at 1/8 of the input resolution without downsampling.
  • Context Extraction Modules: Bilateral fusion exchanges information in both directions between the high- and low-resolution branches, with point-wise summation performed before ReLU.
  • Context Extraction Modules: The networks are offered at different depths and widths, including DDRNet-23-slim, DDRNet-23, DDRNet-39, and a 1.5× wider DDRNet-39.

B. Deep Aggregation Pyramid Pooling Module

DAPPM extracts and hierarchically fuses multi-scale context from low-resolution feature maps. Its richer contextual processing adds little inference cost because it operates at very low spatial resolution.

  • B. Deep Aggregation Pyramid Pooling Module: DAPPM extracts contextual information from low-resolution feature maps using feature aggregation and pyramid pooling.It is applied to feature maps at 1/64 image resolution.
  • B. Deep Aggregation Pyramid Pooling Module: Large pooling kernels with exponential strides generate feature maps at 1/128, 1/256, and 1/512 image resolutions, alongside input and global-average-pooled information.
  • B. Deep Aggregation Pyramid Pooling Module: DAPPM upsamples multi-scale features and fuses them with additional 3×3 convolutions in a hierarchical-residual manner.The resulting features are concatenated and compressed with a 1×1 convolution, with a 1×1 projection shortcut.
  • B. Deep Aggregation Pyramid Pooling Module: DAPPM integrates larger-kernel contexts with deeper information flow, forming multi-scale context from different depths and pooling sizes.
  • B. Deep Aggregation Pyramid Pooling Module: DAPPM provides richer context than PPM while hardly affecting inference speed because its input resolution is only 1/64 of the image resolution.

C. Overall Architecture for Semantic Segmentation

The segmentation architecture modifies a dual-resolution network with deeper low-resolution processing, DAPPM context extraction, low-to-high fusion, and a configurable segmentation head. Training additionally uses an auxiliary loss weighted at 0.4, while the auxiliary head is removed during testing.

  • Overall architecture: The low-resolution branch is further downsampled, and DAPPM extracts contextual information from 1/64-resolution high-level feature maps.These changes adapt the dual-resolution network for semantic segmentation.
  • Overall architecture: The final high-to-low fusion is replaced by low-to-high bilinear interpolation and summation fusion.
  • Overall architecture: The segmentation head uses 3×3 and 1×1 convolutions, with its computational load controlled by the intermediate output dimension.The dimensions are 64 for DDRNet-23-slim, 128 for DDRNet-23, and 256 for DDRNet-39.
  • Deep supervision: Auxiliary supervision is used during training with a loss weight of 0.4, and the auxiliary segmentation head is discarded during testing.The final loss is a weighted sum of the normal and auxiliary cross-entropy losses.

IV. EXPERIMENTS

Experiments cover Cityscapes, CamVid, and COCOStuff under dataset-specific training settings and reported accuracy-speed comparison protocols. Cityscapes provides a large, high-resolution urban-scene benchmark, while the comparison tables specify model complexity and evaluation conditions.

  • Datasets: Cityscapes contains 2975 training, 500 validation, and 1525 test images across 19 classes at 2048×1024 resolution.Only finely annotated data are used for training.
  • Datasets: CamVid contains 701 densely annotated 960×720 frames, with train and validation images merged for training and 11 test classes used for evaluation.
  • Datasets: COCOStuff contains 10K densely annotated images spanning 182 categories, using a 9K/1K train-test split.The categories include 91 thing and 91 stuff classes, with 11 thing classes lacking segmentation annotations.
  • Training: ImageNet pretraining uses 224×224 inputs, batch size 256, 100 epochs, and four 2080Ti GPUs before segmentation finetuning.
  • Training: Cityscapes training uses random crops of 1024×1024, scaling from 0.5 to 2.0, horizontal flipping, and 484 training epochs.
  • Evaluation: The Cityscapes comparison reports accuracy and speed on validation and test sets while documenting GPU, input-resolution, GFLOPs, and TensorRT conditions.

3) COCOStuff:

The reported evaluations use fixed inference protocols and repeated-trial reporting, while the results show strong accuracy-speed performance across the paper’s benchmarks. The method also reaches 80.4% mIoU at 22 FPS on the Cityscapes test server using finely annotated data.

  • Evaluation protocol: Inference speed is measured on a single GTX 2080Ti with batch size 1 under specified CUDA, cuDNN, PyTorch, image-size, and normalization conditions.
  • Evaluation protocol: The experiments report best results, average results, and standard deviations across four trials, except for Cityscapes test accuracy from the official server.
  • Cityscapes: 77.4% mIoU at 102 FPS is achieved by DDRNet-23-slim on the Cityscapes test set.It outperforms DFANet A and MSFNet* by 6.1% mIoU with similar inference speed.
  • Cityscapes: 80.4% mIoU at 22 FPS is achieved by deeper DDRNets on the Cityscapes test server using only finely annotated data.
  • Cityscapes: 79.5% mIoU is achieved by DDRNet-23 on the Cityscapes validation set, while DDRNet-23-slim uses 36.3 GFLOPs and 5.7M parameters.

2) CamVid:

On CamVid, DDRNet-23-slim combines high accuracy with very high throughput, while COCOStuff tests the method under many-category conditions. With test augmentation, DDRNet-39 also improves Cityscapes accuracy and uses substantially less computation than cited comparison models.

  • CamVid: 74.7% mIoU at 230 FPS is achieved by DDRNet-23-slim on the CamVid test set without Cityscapes pretraining.It obtains the second-highest accuracy and runs faster than all other methods.
  • CamVid: DDRNet-23 outperforms MSFNet, BiSeNetV2-L, and SFNet on CamVid while running about two times faster than the latter two methods.
  • COCOStuff: DDRNet-23 achieves accuracy similar to PSPNet50 on COCOStuff while running 20 times as fast.The dataset is described as challenging because it contains plentiful categories.
  • State-of-the-art comparisons: 81.9% mIoU is achieved by DDRNet-39 on Cityscapes after standard multi-scale and horizontal-flip test augmentation, up from 80.4%.It requires 11% of DANet’s computation and 34% of SFNet’s computation.

F. Comparisons with HRNet

DDRNet differs from HRNet in its number of parallel branches and adds multi-scale context extraction to the low-resolution branch. Experiments report improvements over HRNet in inference time and training memory, while training techniques further raise accuracy.

  • Architectural comparison: DDRNet uses fewer parallel branches than HRNet and appends a multi-scale context extraction module to its low-resolution branch.The comparison isolates branch count and the added context module as key architectural differences.
  • Qualitative comparison: Visualizations show successful segmentation examples alongside failures for DDRNet-23-slim and DDRNet-23 on Cityscapes.The figure columns contain the input image, ground truth, and outputs of both models.
  • Architectural comparison: DDRNet improves both inference time and training memory usage over HRNet in comparative experiments.Training memory was measured on a single 2080Ti with batch size 2 and 1024×512 crops, excluding the auxiliary segmentation head.
  • Training techniques: 76.1 to 77.8 accuracy is achieved with deep supervision, OHEM, and training at a larger crop size.These training tricks are also adopted by SFNet according to the supplied passage.

2) DAPPM:

DAPPM improves scene-parsing accuracy by aggregating multi-scale context from low-resolution features with little effect on inference speed. The broader dual-resolution design further improves accuracy at a small time cost while reducing computation and time relative to the baseline.

  • DAPPM: 74.1% mIoU to 77.8% mIoU is obtained with DAPPM, while inference speed is hardly affected.DAPPM is compared with pyramid pooling, self-attention, and res2net-based context extraction modules.
  • DAPPM: DAPPM achieves a 1% mIoU gain over PPM and RES2, whereas Base-OC performs relatively poorly with low-resolution feature maps.RES2 denotes the res2net module and Base-OC denotes the object context module.
  • Qualitative results: Visualizations on CamVid include successful samples and segmentation failures, with ignored labels shown in black.The figure columns contain the input image, ground truth, and DDRNet-23 output.
  • Dual-resolution networks: Bilateral fusions further improve segmentation accuracy at a small time cost.The dual-resolution design provides richer features for DAPPM and the final segmentation head.
  • Dual-resolution networks: The dual-resolution network uses less computation resources and time than the baseline while achieving better performance.The ablation baseline is adapted from BiSeNetV2 by replacing its complicated semantic branch with the low-resolution branch.
  • Conclusion: The method is presented as a simple, efficient baseline that unifies real-time and high-accuracy semantic segmentation without extra bells or whistles.The paper reports a speed–accuracy trade-off across three popular benchmarks and describes future transfer to downstream tasks.
Loading 2101.06085v2…