Source-linked AI summary
ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time
Rudra P K Poudel, Ujwal Bonde, Stephan Liwicki, Christopher Zach
TL;DR
Real-time semantic segmentation requires computational efficiency without sacrificing accuracy, but existing approaches often trade accuracy for speed and memory efficiency. ContextNet addresses this with complementary low-resolution context and full-resolution detail branches, achieving competitive Cityscapes performance while supporting real-time processing.
Problem
Modern semantic segmentation systems achieve high accuracy but are often unsuitable for real-time or embedded applications because reducing computational cost can significantly reduce accuracy.
Method
ContextNet combines a deep low-resolution context branch with a shallow full-resolution detail branch, using depthwise convolutions and bottleneck residual blocks for efficiency.
Results
ContextNet achieves competitive results and outperforms other state-of-the-art real-time methods, reaching 66.1% mIoU at 18.3 fps on the Cityscapes test set.
Takeaways & Limitations
Combining global and local context enables competitive real-time segmentation, while pruning improves segmentation accuracy and ContextNet can be tuned for available resources.
Takeaways & Limitations
The reported Cityscapes experiments use only 5,000 high-label-quality images and no pre-training, while timing comparisons may depend on software environment and testing protocols.
Abstract
from arXiv · showhide
Modern deep learning architectures produce highly accurate results on many challenging semantic segmentation datasets. State-of-the-art methods are, however, not directly transferable to real-time applications or embedded devices, since naive adaptation of such systems to reduce computational cost (speed, memory and energy) causes a significant drop in accuracy. We propose ContextNet, a new deep neural network architecture which builds on factorized convolution, network compression and pyramid representation to produce competitive semantic segmentation in real-time with low memory requirement. ContextNet combines a deep network branch at low resolution that captures global context information efficiently with a shallow branch that focuses on high-resolution segmentation details. We analyse our network in a thorough ablation study and present results on the Cityscapes dataset, achieving 66.1% accuracy at 18.3 frames per second at full (1024x2048) resolution (41.9 fps with pipelined computations for streamed data).
1 Introduction
ContextNet addresses the gap between accurate semantic segmentation and the speed, memory, and energy constraints of real-time and embedded applications. It uses factorized convolutions, compression, and a two-resolution design to combine global context with detailed segmentation.
- Real-time semantic segmentation is important for autonomous driving because applications require accurate road boundaries and obstacle detection.
- Modern segmentation methods achieve high accuracy but often have reduced computational efficiency, limiting direct transfer to real-time applications.
- Factorized convolution, pruning, hashing, and quantization are established techniques for reducing computational cost, memory, or parameter size.
- The architecture targets real-time segmentation with a low memory footprint while retaining competitive accuracy.
- ContextNet uses two resolution branches: a deep low-resolution network captures context, while a shallow high-resolution network refines segmentation boundaries.
2 Proposed Context Network (ContextNet)
ContextNet combines efficient convolutional building blocks with deep low-resolution and shallow full-resolution branches. The branches capture global and local context separately before fusion, enabling faster segmentation while preserving detail.
- ContextNet combines low-resolution global information with a shallow high-resolution sub-network that provides detailed segmentation results.
- Depth-wise separable convolution factorizes standard convolution into depth-wise and 1 × 1 point-wise operations, reducing parameters and floating-point operations.
- Features for local context are extracted from full-resolution input by a shallow branch and combined with low-resolution results.
- The low-resolution branch uses bottleneck residual blocks and 38 efficient layers, while the full-resolution branch contains only four layers for local refinement.
- The fusion unit uses feature addition rather than concatenation, followed by a 1 × 1 convolution for final soft-max classification.
- Larger low-resolution branch inputs improve segmentation but increase runtime, while pyramid pooling and weighted auxiliary loss improve the design empirically.
3 Experiments
The experiments evaluate ContextNet through Cityscapes ablations and benchmark measurements. Performance is reported with mIoU, while runtime includes sequential CPU–GPU transfers and kernel execution.
- The evaluation includes a detailed Cityscapes validation-set ablation study and performance results on the Cityscapes benchmark.
- Experiments use an Nvidia Titan X workstation with CUDA 8.0 and cuDNN V6, with batch normalization merged into inference layers.
- Segmentation results are reported as mean intersection-over-union, while runtime includes CPU–GPU memory transfers and kernel execution.
3.2 Cityscapes Dataset
The Cityscapes experiments use only the dataset’s high-quality annotations and compare ContextNet configurations across resolution, branching, and parameter-count choices. The validation protocol is based on fixed training and validation splits without pre-training.
- Cityscapes contains street-scene images from 50 German cities, including 5,000 images with high pixel-accuracy labels.
- Table 4 compares quarter-, half-, and eighth-resolution context branches, multiple-resolution branches, and reduced-memory variants on Cityscapes validation.
- The experiments use 2,975 training images, 500 validation images, and 1,525 test images from the high-quality subset.
- Ablation weights are learned solely from the Cityscapes training set and evaluated on its validation set, with no pre-training.
Ablation Study
ContextNet’s deeper low-resolution context branch improves segmentation accuracy, but higher input resolution increases runtime.
- Larger resolution in the deeper context branch produces better segmentation results.
- Improved accuracy from larger context-branch resolutions comes at the cost of running time.
- 18.3 fps and 65.9% mIoU make cn14 a balanced choice for segmentation performance and speed.
- Higher resolution benefits small-size, fine-detail, and rare classes, while larger classes can be captured at lower resolution.
Multiple Branches:
Adding multiple resolutions can improve accuracy, but the two-branch design offers a better runtime trade-off in ContextNet’s implementation.
- Multiple Branches:: A two-shallow-branch cn124 improves accuracy by 1.4% over cn14 but reduces runtime from 18.3 fps to 7.6 fps.
- Multiple Branches:: cn12 outperforms cn124 in both accuracy and speed, indicating that a deeper higher-resolution sub-network is more beneficial.
- Multiple Branches:: The detailed IoU comparison covers cn14, cn12, cn18, and the multi-level cn124 variant, with high resolution benefiting several class groups.
- Number of Parameters:: ContextNet variants with 159,387 and 490,459 parameters achieve 57.7% and 62.1% mIoU, respectively.
Number of Parameters:
Independent context and detail branches capture different aspects of segmentation, but their separate ensemble underperforms the integrated ContextNet architecture.
- ContextNet vs Ensemble Nets:: 63.3% and 25.1% mIoU were obtained by independently trained global-context and detail branches, respectively.
- ContextNet vs Ensemble Nets:: The independently trained branch ensemble reaches 60.3% mIoU, 6.6% below cn14 and 3% below the context branch alone.
- ContextNet vs Ensemble Nets:: The context branch detects larger objects but fails around boundaries and thin regions, whereas the detail branch detects boundaries but misses object centers.
- ContextNet vs Ensemble Nets:: ContextNet combines the complementary information learned by the context and detail branches.
Context Branch Analysis:
ContextNet combines global context and high-resolution detail to improve segmentation, including on small objects, while maintaining real-time performance.
- Context Branch Analysis:: 65.9% mIoU is achieved by ContextNet, compared with 63.3% for the quarter-resolution context branch and 25.1% for the independent detail branch.
- Context Branch Analysis:: The context branch captures larger objects, while the detail branch captures boundaries and thin regions.
- Context Branch Analysis:: ContextNet runs at 18.3 fps and reaches 66.1% mIoU after pruning on the Cityscapes test set.
Cityscape Benchmark Results
ContextNet is evaluated against real-time and offline semantic-segmentation methods on Cityscapes, with benchmark accuracy and runtime reported under distinct evaluation conditions.
- Benchmark accuracy and runtime: 69.5% and 68.0% mIoU are reported for ERFNet and ICNet, respectively, but both are considerably slower than ContextNet.
- Runtime conditions: 41.9 fps is achieved when memory transfer and kernel execution are pipelined for streamed data.The baseline runtime evaluation includes the complete CPU and GPU pipeline, including memory transfers.
Network Pruning:
The paper notes a timing discrepancy between its ICNet implementation and the runtime reported by the original authors.
- Network Pruning:: The authors’ ICNet implementation achieves similar accuracy but does not reproduce the reported timing.They attribute this difference possibly to software environment and testing-protocol differences.
4 Conclusions and Future Work
The paper concludes that ContextNet combines low-resolution context with high-resolution detail for real-time segmentation and benefits from pruning, while broader autonomous-system applications remain future work.
- 4 Conclusions and Future Work: ContextNet combines a deep low-resolution branch with a shallow full-resolution branch to improve segmentation details efficiently.The architecture also leverages depthwise convolutions and bottleneck residual blocks for memory and runtime efficiency.
- 4 Conclusions and Future Work: The ablation study finds that ContextNet combines global and local context competitively and outperforms other state-of-the-art real-time methods.
- 4 Conclusions and Future Work: Model pruning improves segmentation accuracy while targeting specified network-size and real-time-performance requirements.
- 4 Conclusions and Future Work: Single-image depth prediction is identified as future work for assessing ContextNet on other autonomous-system tasks.