Source-linked AI summary
In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Images
Marin Oršić, Ivan Krešo, Petra Bevandić, Siniša Šegvić
TL;DR
Real-time road-driving segmentation must process large images with low latency despite high computational demands and the limitations of custom lightweight architectures. The paper combines lightweight ImageNet-grade encoders, lateral-connection upsampling, and shared multi-resolution fusion. SwiftNetRN-18 achieves 75.5% mIoU at 39.9 Hz on full-resolution Cityscapes images using a GTX1080Ti.
Problem
Real-time semantic segmentation requires high-resolution predictions and very low latency, while many lightweight approaches use custom architectures that miss transfer-learning regularization and risk overfitting.
Method
The method uses compact ImageNet-grade encoders, lightweight decoders with lateral connections, and spatial pyramid pooling or shared-parameter pyramid fusion.
Results
75.5% mIoU at 39.9 Hz is achieved for 1024×2048 Cityscapes images on a GTX1080Ti by the SwiftNetRN-18 submission.
Takeaways & Limitations
The reported results support a strong speed-accuracy trade-off from compact ImageNet-oriented encoders and lightweight lateral-connection decoders for real-time segmentation.
Takeaways & Limitations
The method assumes an ImageNet-pretrained encoder to benefit from transfer-learning regularization, while also requiring a decoder that restores prediction resolution without losing real-time speed.
Abstract
from arXiv · showhide
Recent success of semantic segmentation approaches on demanding road driving datasets has spurred interest in many related application fields. Many of these applications involve real-time prediction on mobile platforms such as cars, drones and various kinds of robots. Real-time setup is challenging due to extraordinary computational complexity involved. Many previous works address the challenge with custom lightweight architectures which decrease computational complexity by reducing depth, width and layer capacity with respect to general purpose architectures. We propose an alternative approach which achieves a significantly better performance across a wide range of computing budgets. First, we rely on a light-weight general purpose architecture as the main recognition engine. Then, we leverage light-weight upsampling with lateral connections as the most cost-effective solution to restore the prediction resolution. Finally, we propose to enlarge the receptive field by fusing shared features at multiple resolutions in a novel fashion. Experiments on several road driving datasets show a substantial advantage of the proposed approach, either with ImageNet pre-trained parameters or when we learn from scratch. Our Cityscapes test submission entitled SwiftNetRN-18 delivers 75.5% MIoU and achieves 39.9 Hz on 1024x2048 images on GTX1080Ti.
1. Introduction
Real-time semantic segmentation must combine high-resolution perception with low latency, creating severe computational demands. The paper addresses this trade-off with lightweight ImageNet-grade architectures, efficient upsampling, and shared multi-resolution features.
- Motivation: Semantic segmentation predicts class posteriors for every pixel, while autonomous-navigation applications require large images and very low latency.These requirements intensify computational strain and make real-time implementations challenging.
- Limitations of prior approaches: Most real-time approaches use custom lightweight architectures and often train from scratch, missing transfer-learning regularization and risking overfitting.ImageNet pre-training can help, but the reported benefits tend to be smaller for architectures not designed for competitive ImageNet performance.
- Approach: The proposed models use lightweight ImageNet-grade classification architectures as recognition engines, lightweight lateral-connection decoders, and shared-resolution pyramidal fusion.The design targets a competitive blend of efficiency and prediction accuracy across computing budgets.
- Results: The experiments report state-of-the-art semantic-segmentation accuracy among approaches targeting real-time execution, using both ImageNet pre-training and training from scratch.The paper evaluates models on several road-driving datasets and computing budgets.
2. Related Work
Prior real-time segmentation methods reduce computation through custom encoders, downsampling strategies, trained upsampling, and receptive-field modules. The paper instead emphasizes ImageNet-competitive encoders and a different multi-resolution fusion strategy.
- Resolution restoration: Avoiding downsampling preserves resolution but increases latent-representation cost, while dilated convolutions add implementation overhead.These approaches can therefore conflict with real-time efficiency.
- Upsampling decoders: Encoder-decoder models recover input resolution by upsampling subsampled representations, often blending deep semantic features with shallow spatial features through lateral connections.The upsampling path must remain lean enough for efficiency without sacrificing necessary capacity.
- Receptive-field enlargement: Spatial pyramid pooling enlarges receptive fields by aggregating features over grids with different granularities, but large-capacity variants may hurt generalization.The paper studies resolution pyramids as an alternative that also promotes scale invariance.
- Pyramidal fusion: Earlier pyramidal segmentation methods mainly fuse deepest representations, whereas this work combines different abstraction levels before the decoder upsampling path.The proposed arrangement is intended to improve gradient flow throughout the network.
- Custom lightweight encoders: Real-time systems commonly use custom encoders such as ICNet or ERFNet rather than encoders designed for competitive ImageNet recognition.ERFNet reduces parameters by replacing a residual-block convolution with sequential 3×1 and 1×3 convolutions.
- Closest related methods: Related work using MobileNet V2 and NASNet employs thicker lateral-connection decoders, whereas this method uses a lower-capacity decoder and exclusively lateral-connection upsampling.The paper reports a 4-fold speed-up from avoiding dilated convolutions and a 5 percentage-point Cityscapes-test improvement from full-resolution training.
3. The proposed segmentation method
The proposed segmentation system combines compact recognition encoders, ladder-style lateral upsampling, and receptive-field enlargement through spatial pyramid pooling or shared-parameter pyramid fusion.
- Design requirements: The method assumes ImageNet-pretrained encoders, resolution-restoring decoders, simple upsampling, and gradient flow sufficient for training from random initialization.These requirements target transfer-learning benefits, detailed predictions, real-time speed, and trainability.
- Recognition encoder: ResNet-18 and MobileNet V2 provide compact recognition encoders suited to fine-tuning, training from scratch, and real-time operation.Although ResNet-18 is around six times more complex computationally, MobileNet V2 can be slower because depthwise separable convolutions lack direct cuDNN support.
- Upsampling decoder: The decoder upsamples coarse semantic features with bilinear interpolation, sums them with lateral encoder features, and blends the result using a 3×3 convolution.Lateral features are routed from the elementwise sum within the last residual block at the corresponding subsampling level.
- Single scale model: The single-scale model uses a downsampling encoder, spatial pyramid pooling, and a lightweight decoder to produce dense predictions at the original resolution.Encoder features reach H/32×W/32 before spatial pyramid pooling and decoder upsampling.
- Single scale model: The decoder is asymmetric: it has one convolution per upsampling module, constant feature dimensionality, and 1×1 lateral projections to adjust encoder-feature dimensions.Each module performs bilinear upsampling, elementwise summation with the lateral connection, and 3×3 convolutional blending.
- Interleaved pyramid fusion: Compact encoders reduce capacity and receptive field, so image pyramids are used to enlarge the receptive field while reducing capacity requirements.This addresses limitations introduced by lightweight recognition backbones.
- Interleaved pyramid fusion: Interleaved pyramid fusion applies shared encoder parameters at multiple image resolutions and concatenates equal-resolution features before bottlenecking and decoder blending.Shared parameters support recognition of differently sized objects with a common parameter set and promote lateral connections and gradient flow.
4. Experiments
The experiments evaluate accuracy, computational complexity, and execution speed on Cityscapes and CamVid using desktop and embedded GPUs. Measurements combine platform-agnostic operation counts with hardware-specific end-to-end throughput.
- Evaluation setup: Experiments cover Cityscapes and CamVid, reporting mIoU, GFLOP, and execution speed on GTX1080Ti and Jetson TX2 hardware.Ablation and validation experiments analyze the effects of design choices.
- Efficiency metrics: GFLOP counts fused multiply-add operations and provides a platform-agnostic complexity measure, especially for CPUs where multiplications have roughly similar costs.The paper notes that GFLOP does not necessarily track actual processing speed.
- Efficiency metrics: FPS measures processing time on a particular platform and is reported with batch size 1 to simulate real-time applications.The timing interval runs from transferring input data to the GPU through receiving predictions in RAM.
- Measurement protocol: Measurements average FPS over 1000 forward passes on a GTX1080Ti, excluding batch-normalization layers because they would be fused with preceding convolutions in real-time use.The setup uses CUDA 10.0, cuDNN 7.3, and PyTorch 1.0rc1.
4.3. Cityscapes
On Cityscapes, the single-scale ResNet-18 model combines high segmentation accuracy with real-time inference at full image resolution. ImageNet pre-training is important for maintaining this performance.
- 75.5% test mIoU and 39.9 FPS are achieved on full Cityscapes resolution of 1024×2048 pixels with the single-scale ResNet-18 model.The submission is titled SwiftNetRN-18 and is reported to outperform other approaches targeting real-time operation.
- 5 mIoU percentage points separate ImageNet-pre-trained models from corresponding models trained from scratch.The result identifies ImageNet initialization as an important ingredient for highly accurate predictions.
- Custom encoders such as ERFNet gain only 1.7 mIoU percentage points from ImageNet pre-training.
- The pyramid model produces no measurable improvement over the single-scale model on Cityscapes.
4.4. CamVid
On CamVid, the pyramid model improves segmentation accuracy, while ImageNet pre-training is especially valuable given the dataset’s much smaller pixel count. The experiments compare pre-trained and scratch-trained models on 11 classes.
- Roughly 1.5 pp mIoU improvement is obtained with the pyramid model using pre-trained ResNet-18 and MobileNetV2 backbones.The improvement is reported on the CamVid test subset.
- CamVid contains 701 densely annotated frames and is evaluated across 11 semantic classes.The experiment uses 367 training, 101 validation, and 233 test images, combining train and validation for training.
- 7-9pp of mIoU performance is contributed by ImageNet pre-training on CamVid, more than on Cityscapes.CamVid has almost 20 times fewer training pixels than Cityscapes.
- The pyramid model achieves better results than the single-scale model on CamVid.This supports sharing encoder parameters across pyramid levels.
4.5. Validation of the upsampling capacity
The upsampling path’s feature-map capacity is validated on Cityscapes using ImageNet initialization. Accuracy saturates at 128 dimensions, which is selected as the speed–accuracy trade-off.
- 128 dimensions is where model accuracy saturates for feature maps along the upsampling path.The models are trained on Cityscapes train images at 512×1024 and evaluated on Cityscapes validation images.
- The number of feature maps along the upsampling path is identified as the decoder’s most important design choice.
- 128 dimensions is used in subsequent experiments as a sensible speed-accuracy trade-off.
4.6. Ablation of lateral connections
Lateral connections materially improve Cityscapes validation accuracy, while execution profiling separates encoder/SPP and decoder costs. The decoder is substantially faster than the ResNet-18 encoder.
- Removing lateral connections lowers validation accuracy from 75.35% to 72.93% on full Cityscapes training images.The ablation discards 1×1 skip-connection convolutions and elementwise summations in upsampling modules.
- The decoder is twice as fast as the ResNet-18 encoder in the single-scale model.Execution times and GFLOPs are reported separately for the downsampling and upsampling paths at 2048×1024 resolution.
- ResNet-18 is almost twice as fast as MobileNet v2 despite requiring 6 times more multiplications.The discrepancy is observed in the downsampling paths, including the encoder and SPP.
4.8. Size of the receptive field
The effective receptive field is measured from central-pixel gradients on Cityscapes val, using the standard deviation of pixels in the top 5% of gradients. Both SPP and interleaved pyramidal fusion substantially increase the receptive field.
- Both SPP and interleaved pyramidal fusion substantially increase the effective receptive field.
- Effective receptive fields are measured as the standard deviation of pixels with top 5% image gradients for the dominant central-pixel class.The measurement uses gradients of the central pixel’s logits with respect to the input image.
4.9. Processing speed on Jetson TX2
The proposed architecture is evaluated on an NVIDIA Jetson TX2 across two backbones and multiple input resolutions. MobileNet V2 is generally faster, while ResNet-18 leads at the lowest resolution.
- MobileNet V2 outperforms ResNet-18 by 20-30% on most resolutions because it uses fewer FLOPs.
- ResNet-18 is faster than MobileNet V2 at the lowest resolution.
- The implementations do not use TensorRT optimizations, and bilinear interpolation is replaced with nearest-neighbour interpolation because of limited CUDA cores.
- Processing speed is reported in frames per second on an NVIDIA Jetson TX2 for two backbones and various input resolutions.
5. Conclusion
The paper argues that compact ImageNet-oriented encoders and lightweight decoders provide a strong speed-accuracy trade-off for real-time semantic segmentation. Its Cityscapes submission achieves 75.5% mIoU at 39.9 Hz on full-resolution images.
- Compact ImageNet-oriented encoders with lightweight lateral-skip decoders are presented as a better speed-accuracy trade-off for real-time segmentation.
- Interleaved pyramidal fusion further improves results on large objects close to the camera.
- 75.5% mIoU is achieved while processing 1024×2048 images at 39.9 Hz on a GTX1080Ti.