PaperlayerView exact revision on arXiv

Source-linked AI summary

ICNet for Real-Time Semantic Segmentation on High-Resolution Images

Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, Jiaya Jia

arXiv:1704.08545v2cs.CV

TL;DR

Real-time semantic segmentation must reduce dense pixel-wise inference cost without sacrificing too much quality, particularly for high-resolution images. ICNet uses low-, medium-, and high-resolution branches with cascade feature fusion and label guidance to progressively refine predictions. It delivers real-time performance on challenging datasets, including 30 fps at 1024 × 2048 on one TitanX GPU, while reporting substantial speed and memory savings.

  • Problem

    High-resolution semantic segmentation requires dense pixel-wise inference, but improving accuracy generally increases operations and existing fast methods can sacrifice quality.

  • Method

    ICNet combines low-, medium-, and high-resolution image branches with cascade feature fusion and cascade label guidance to refine coarse predictions progressively.

  • Results

    ICNet achieves 69.5% mIoU and 30 fps on 1024 × 2048 images using one TitanX GPU, with training on fine and coarse data raising mIoU to 70.6%.

  • Takeaways & Limitations

    The system provides a practical speed–accuracy balance for real-time high-resolution segmentation across Cityscapes, CamVid, and COCO-Stuff.

  • Takeaways & Limitations

    Reported inference speed uses a single network forward, while some mIoU comparisons may include multi-scale and flipping test-time tricks that require more time.

Abstract

from arXiv · show

We focus on the challenging task of real-time semantic segmentation in this paper. It finds many practical applications and yet is with fundamental difficulty of reducing a large portion of computation for pixel-wise label inference. We propose an image cascade network (ICNet) that incorporates multi-resolution branches under proper label guidance to address this challenge. We provide in-depth analysis of our framework and introduce the cascade feature fusion unit to quickly achieve high-quality segmentation. Our system yields real-time inference on a single GPU card with decent quality results evaluated on challenging datasets like Cityscapes, CamVid and COCO-Stuff.

1 Introduction

Real-time semantic segmentation must balance dense-prediction accuracy against substantial computation, especially for high-resolution images. ICNet addresses this tension with a multi-resolution cascade that combines low-resolution efficiency with high-resolution detail.

  • 1 Introduction: Pixel-level semantic segmentation accuracy typically increases with operations, making inference speed a central challenge for high-resolution inputs.High-accuracy ResNet38 and PSPNet require around 1 second for a 1024 × 2048 image, while faster ENet and SQ achieve mIoUs below 60%.
  • 1 Introduction: The paper reports a 5× inference speedup and 5× lower memory consumption while retaining high-quality results at 1024 × 2048 resolution.The authors position ICNet as a practical balance between efficiency and prediction accuracy.
  • 1 Introduction: ICNet uses low-, medium-, and high-resolution image branches with cascade feature fusion and cascade label guidance.The low-resolution branch produces coarse semantic information, while later branches progressively integrate higher-resolution details.

2 Related Work

Prior semantic segmentation research improved representation quality through CNNs, dilated convolutions, encoder-decoder designs, multi-scale features, and spatial-relationship modeling. However, many effective architectures preclude real-time inference, while lightweight methods sacrifice accuracy.

  • 2 Related Work: CNN-based segmentation methods improved performance through fully convolutional networks and increasingly sophisticated architectures.Representative designs include dilated convolutions, encoder-decoder structures, multi-scale ensembles, CRF or MRF modeling, pyramid pooling, and wider networks.
  • 2 Related Work: Many effective segmentation architectures remain incompatible with real-time inference because their designs require substantial computation.The related work includes multi-path refinement, pyramid pooling, wide networks, and spatial-relationship models.
  • 2 Related Work: Lightweight methods such as ENet and SQ improve efficiency but notably sacrifice accuracy.The paper contrasts these methods with high-accuracy approaches that are slower on high-resolution images.
  • 2 Related Work: Video-specific approaches reduce computation by reusing or propagating features across frames.Clockwork, Deep Feature Flow, FSO, and NetWarp exploit temporal redundancy, optical flow, or feature warping.

3 Image Cascade Network

ICNet combines multi-resolution image branches with cascade feature fusion and label guidance to reduce computation while refining high-resolution segmentation.

  • Architecture: ICNet analyzes resolution-dependent computation and uses low-resolution processing to obtain coarse semantic predictions efficiently.Convolutional cost depends strongly on feature-map resolution, while the low-resolution branch limits computation even with a deep backbone.
  • Architecture: The network receives low-, medium-, and high-resolution versions of the input, with higher-resolution branches recovering and refining coarse predictions.The full-resolution input is downsampled by factors of 2 and 4 for the medium- and high-resolution branches.
  • Efficiency: The three branches require 18 ms, 6 ms, and 9 ms for the low-, medium-, and high-resolution processing stages, respectively, while sharing computation between the first two.The architecture is described as efficient and memory friendly while preserving good-quality segmentation.
  • Cascade Feature Fusion: Cascade feature fusion upsamples F1, refines it with dilated convolution, projects F2, and combines the processed features by normalization, summation, and ReLU.The fusion unit aligns F1 and F2 spatially and channels before producing the fused feature.
  • Cascade Label Guidance: Cascade label guidance trains the low-, medium-, and high-resolution branches with ground-truth labels at corresponding scales and weighted losses.During testing, low- and medium-resolution guidance operations are removed, retaining only the high-resolution branch.

4 Structure Comparison and Analysis

ICNet differs from prior segmentation architectures by assigning heavy computation to the lowest-resolution input and using higher-resolution inputs for progressive refinement.

  • Structure Comparison: Unlike prior high-resolution frameworks, ICNet feeds only the lowest-resolution input into the heavy CNN for coarse semantic prediction.Higher-resolution inputs then recover and refine the prediction progressively.

5 Experimental Evaluation

The evaluation targets high-resolution semantic segmentation across Cityscapes, CamVid, and COCO-Stuff using controlled single-GPU experiments.

  • Datasets: ICNet is evaluated on Cityscapes, CamVid, and COCO-Stuff, including images up to 1024 × 2048, 720 × 960, and 640 × 640, respectively.COCO-Stuff is emphasized because its larger images make real-time performance more difficult than on many lower-resolution datasets.
  • Evaluation Setup: Experiments use Caffe on a single Maxwell TitanX GPU, with inference time averaged over 100 repeated iterations.Batch-normalization parameters are merged into neighboring convolution layers for testing.
  • Evaluation Setup: Training uses mini-batches of 16, a base learning rate of 0.01, poly scheduling, momentum 0.9, weight decay 0.0001, and random mirror and resizing augmentation.Maximum iterations are 30K for Cityscapes, 10K for CamVid, and 30K for COCO-Stuff.
  • Datasets: Cityscapes provides 5,000 finely annotated 1024 × 2048 images, with 2,975 training, 500 validation, and 1,525 testing images.The dataset contains 30 common classes, with 19 used for training and testing.

Intuitive Speedup

The paper evaluates input and feature downsampling plus model compression, finding that intuitive speedups trade substantial segmentation quality for insufficient real-time performance.

  • Speedup Strategies: The evaluated intuitive speedup strategies are input downsampling, feature downsampling, and model compression.These strategies target the main contributors to network complexity and inference time.
  • Downsampling Input: Input downsampling accelerates inference but produces coarse maps, misses small objects, blurs boundaries, or remains too slow for real-time use.The 0.25 scaling ratio loses detail, while the 0.5 ratio still misses distant people and traffic lights.
  • Downsampling Feature: Feature downsampling makes inference faster while sacrificing accuracy because low-level detail is lost, and even a 1:32 feature map remains insufficiently fast.Table 1 compares feature downsampling factors 8, 16, and 32.
  • Model Compression: A half-compressed PSPNet50 baseline takes 170 ms and reaches 67.9% mIoU, indicating compression alone does not achieve real-time performance with decent quality.The paper uses this baseline for cascade-branch ablations.

Cascade Branches

ICNet combines low-, medium-, and high-resolution branches through cascade feature fusion and label guidance. Adding branches substantially improves Cityscapes validation quality with small runtime increases.

  • Branch configurations: The ‘sub4’, ‘sub24’, and ‘sub124’ settings use one, two, and three resolution branches, respectively.‘sub4’ uses only the low-resolution input; ‘sub24’ adds medium-resolution information, while ‘sub124’ includes all three branches.
  • Validation results: 67.7% mIoU is achieved with three branches, compared with 66.5% using two branches and 59.6% using only the low-resolution branch.These results are reported on the Cityscapes validation set.
  • Validation results: Adding the second and third branches increases runtime by only 7ms and 8ms, respectively.The reported increases accompany the improvements from the low-resolution baseline.
  • Efficiency: ICNet’s segmentation quality nearly matches the baseline while running 5.2× faster and using 5.8× less memory.The comparison is reported for the tested Cityscapes configuration.
  • Ablation findings: The cascade feature fusion unit improves mIoU over 3×3 and 5×5 deconvolution at similar efficiency, while matching 7×7 deconvolution more closely but processing faster.Removing cascade label guidance lowers performance.

Methods Comparison

ICNet is evaluated against methods spanning accuracy-focused and speed-focused semantic segmentation. It reaches 69.5% mIoU and 30fps on high-resolution Cityscapes images while retaining substantially higher quality than the fastest baselines.

  • Visual comparison: The visual branch analysis compares prediction differences between ‘sub24’ and ‘sub4’, and between ‘sub124’ and ‘sub24’.The figure captions identify white regions as prediction differences for these branch pairs.
  • Evaluation protocol: Cityscapes leaderboard comparisons report mIoU and inference time while excluding methods without reported running time.Some methods may use time-consuming multi-scale testing for best quality.
  • Cityscapes results: ICNet achieves 69.5% mIoU on Cityscapes and is about 10 points higher than ENet and SQ.Training with both fine and coarse data raises mIoU to 70.6%.
  • Cityscapes results: ICNet runs at 30fps on 1024 × 2048 images using one TitanX GPU card.The reported result is for the Cityscapes test setting.

Visual Improvement

ICNet progressively refines coarse predictions as medium- and high-resolution branches recover details. Quantitative connected-component analysis shows the largest gains for small regions.

  • Dataset evaluations: The CamVid and COCO-Stuff result tables report dataset-specific test outcomes, using 720 × 960 and 640×640 evaluation resolutions.The supplied captions identify the reported resolutions but not individual metric values.
  • Visual Improvement: The low-resolution ‘sub4’ branch captures most semantically meaningful objects but misses small regions such as poles and traffic signs.Its predictions are coarse because the input resolution is low.
  • Visual Improvement: The ‘sub24’ branch re-estimates many missed regions, while ‘sub124’ further refines distant objects and blurry boundaries.The full-resolution branch produces the best output among the visualized settings.
  • Quantitative Analysis: Connected-component analysis measures each region’s accuracy as correctly predicted pixels divided by the region’s total pixels.Regions are grouped by size into a histogram to compare branch improvements.
  • Quantitative Analysis: The histogram uses bin size 30 and interval K=3,000, covering region sizes from 1 to 90K.Regions exceeding 90K are ignored.
  • Quantitative Analysis: The largest accuracy differences between successive branches occur in the histogram’s front bins, corresponding to small regions.The blue histogram compares ‘sub24’ with ‘sub4’, and the green histogram compares ‘sub124’ with ‘sub24’.

5.3 CamVid

CamVid contains high-resolution video-derived images and uses an 11-class evaluation split. ICNet delivers real-time inference while maintaining high-quality segmentation on the test set.

  • Dataset: CamVid images have resolution up to 720 × 960, and its evaluation uses 11 semantic classes.The adopted split contains 367 training, 100 validation, and 233 testing images.
  • Results: ICNet reaches 27.8fps on the CamVid test set, 5.7× faster than the second-fastest method and 5.1× faster than the basic model.Inference is measured at the dataset’s high resolution.
  • Results: ICNet achieves high-quality segmentation on CamVid in addition to its higher inference efficiency.The passage reports visual results in supplementary material.
  • Cross-dataset comparison: On COCO-Stuff, ICNet is reported as more efficient and accurate than FCN and DeepLab, with a 5.4× speedup over the baseline.COCO-Stuff evaluates common thing and stuff understanding across up to 182 classes.

6 Conclusion

ICNet targets real-time semantic segmentation by balancing inference speed and prediction accuracy. This balance can make segmentation more practical for tasks requiring fast scene and object segmentation.

  • ICNet accelerates network inference without sacrificing much performance through a multi-resolution framework and a feature fusion unit.These are identified as the system’s major contributions for reducing operations while maintaining quality.
  • The system’s speed–accuracy balance can benefit tasks requiring fast scene and object segmentation.The paper connects this balance to broader practicality for semantic segmentation in other disciplines.
  • ICNet enhances the practicality of semantic segmentation in disciplines where fast scene and object segmentation is needed.
Loading 1704.08545v2…