Source-linked AI summary
Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascade
Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, Xiaoou Tang
TL;DR
Deep semantic segmentation needs both high accuracy and practical speed, but increasing model capacity raises runtime complexity. The paper proposes Deep Layer Cascade, which routes pixels by difficulty through jointly trained stages and computes convolutions only on forwarded regions. LC achieves state-of-the-art performance on PASCAL VOC and Cityscapes while running in real time with competitive accuracy.
Problem
Deep segmentation models improve accuracy but their runtime complexity impedes deployment in applications requiring real-time performance.
Method
Deep Layer Cascade converts one deep model into jointly trained stages that reject easy regions early and progressively forward harder regions for region-specific convolution.
Results
LC achieves state-of-the-art performance on PASCAL VOC and Cityscapes and improves IRNet speed by 42.8% and accuracy by 1.7%.
Takeaways & Limitations
LC provides a difficulty-aware framework that reduces computation while preserving competitive segmentation accuracy and enabling real-time operation.
Takeaways & Limitations
The study leaves 30% of HS pixels with wrong annotations uncorrected to enable fair comparison with previous work.
Abstract
from arXiv · showhide
We propose a novel deep layer cascade (LC) method to improve the accuracy and speed of semantic segmentation. Unlike the conventional model cascade (MC) that is composed of multiple independent models, LC treats a single deep model as a cascade of several sub-models. Earlier sub-models are trained to handle easy and confident regions, and they progressively feed-forward harder regions to the next sub-model for processing. Convolutions are only calculated on these regions to reduce computations. The proposed method possesses several advantages. First, LC classifies most of the easy regions in the shallow stage and makes deeper stage focuses on a few hard regions. Such an adaptive and 'difficulty-aware' learning improves segmentation performance. Second, LC accelerates both training and testing of deep network thanks to early decisions in the shallow stage. Third, in comparison to MC, LC is an end-to-end trainable framework, allowing joint learning of all sub-models. We evaluate our method on PASCAL VOC and Cityscapes datasets, achieving state-of-the-art performance and fast speed.
1. Introduction
Deep semantic segmentation models improve accuracy but incur runtime costs that hinder real-time applications. Deep Layer Cascade (LC) addresses this by routing pixels according to difficulty, reducing computation while improving accuracy.
- Motivation: Deeper architectures improve segmentation accuracy but increase runtime complexity, limiting deployment in applications requiring real-time performance.ResNet-101 is six times deeper than VGG-16 and outperforms it by 4 percent on PASCAL VOC 2012.
- Motivation: Pixels are partitioned into easy, moderate, and extremely hard sets, with easy pixels covering at least 30% of most objects.About 70% of extremely hard pixels lie on ambiguous object boundaries.
- Method: LC treats different layers of one deep network as cascade stages, allowing later layers to focus on a small set of challenging pixels.Unlike model cascade, LC jointly trains all stages rather than fixing earlier stages during later-stage learning.
- Method: LC propagates low- and moderate-confidence pixels forward while rejecting high-confidence easy pixels and extremely hard pixels at each stage.Extremely hard boundary pixels are discarded because fitting them during training may cause overfitting at test time.
- Results: At least 30% of pixels are easy, and rejecting easy and extremely hard pixels reduces computations by more than 40% in earlier stages while improving accuracy.Background pixels comprise 72.5%, enabling deeper layers to focus on foreground objects.
- Results: Applying LC to Inception-ResNet-v2 improves speed by 42.8% and accuracy by 1.7%.The approach is presented as applicable to several advanced network structures.
2. Related Work
Prior segmentation work uses deep CNNs for accuracy and lightweight architectures for speed, while cascade approaches process difficult cases progressively. LC adapts this cascade idea to deep segmentation networks and reports improved performance for IRNet-LC.
- Semantic Image Segmentation: Deep CNNs substantially improve semantic segmentation accuracy by learning stronger representations from modern architectures.The related work includes fully convolutional transformations and other deep segmentation models.
- Efficient Architectures: Lightweight networks reduce parameters and convolutional computation, but prior approaches sacrifice the high performance of deeper models.SegNet removes unnecessary layers, while ENet uses bottleneck modules.
- Layer Cascade: Figure 2 contrasts the Inception-ResNet-v2 classification architecture with its Layer Cascade adaptation, IRNet-LC.The figure also presents IRNet component structures.
- Network Cascades: Cascade methods route difficult cases to later expert networks, but previous network cascades train each network separately.Examples include CNN cascades for face detection and cascaded regression for human pose estimation.
3. Deep Layer Cascade (LC)
Deep Layer Cascade converts a single deep segmentation network into three jointly trained stages that progressively process harder regions. Region convolution restricts later computations to forwarded pixels, while the threshold controls the speed–accuracy trade-off.
- From IRNet to LC: LC divides IRNet into three stages, appending convolutional layers and a softmax loss to each stage.Stage-1 ends at Reduction-A, stage-2 lies between Reduction-A and Reduction-B, and the remaining layers form stage-3.
- Difficulty-aware cascade: Each stage predicts a label map, accepts pixels whose maximum confidence exceeds ρ, and forwards the remaining pixels to the next stage.Stage-2 repeats the same forwarding procedure for stage-3.
- Threshold control: ρ = 0.9 discards 52% of pixels in stage-1 and 35% in stage-2, whereas ρ = 1.0 recovers the original IRNet.Smaller values than 0.9 discard more moderate pixels and can hinder performance.
- Experimental behavior: ρ = 0.95 yields 18 FPS versus 9 FPS for IRNet and improves VOC validation accuracy by 0.8%; ρ = 0.985 yields 15 FPS and a 1.7% improvement.The reported robustness range is ρ ∈[0.9, 1.0].
- Region convolution: Later stages apply region convolution only to forwarded regions, setting other locations to zero and reducing computation.In residual modules, masked outputs are added to identity features, so each stage learns features for its assigned region.
- Training: All stages are jointly optimized with pixel-wise softmax losses, and LC supervision differs by stage to support adaptive processing of easy and hard regions.The training procedure includes initial training followed by cascade training.
4. Experiments
Experiments show that Layer Cascade adaptively routes regions to different stages, improving segmentation accuracy while reducing computation and preserving a favorable speed–performance trade-off. Across ablations and benchmarks, LC outperforms comparable approaches and generalizes to VOC12 and Cityscapes.
- Ablation Study: LC separates easy, moderate, and extremely hard regions using pixel-wise softmax confidence for adaptive stage processing.The threshold ρ controls which regions are discarded early and which are propagated to deeper stages.
- Ablation Study: ρ = 0.985 processes around 52% of regions in early stages and achieves the best performance.Lower thresholds increase early processing but can harm accuracy when hard regions receive insufficient deep inference.
- Ablation Study: Deep supervision provides only a 0.48 mIoU gain over IRNet, while Model Cascade performs worse because its independent sub-models weaken overall modeling capacity.LC retains the whole model depth and adaptively applies it to hard regions.
- Stage-wise Analysis: Stage-wise analysis shows deeper stages focus on harder classes such as bicycle, chair, and dining table, while easy regions are labeled earlier.The analysis reports that harder classes have more pixels handled by stage-3.
- Performance and Speed Analysis: 73.91 mIoU at 14.7 FPS gives LC the best performance among the compared methods while avoiding a large speed–accuracy trade-off.At 23.6 FPS, LC still achieves 66.95 mIoU, showing an adjustable performance–speed operating point.
- Benchmark: LC achieves 80.3 mIoU on VOC12, improving to 82.7 with COCO pre-training, and reaches 71.1 mIoU on Cityscapes.The method wins 16 of 20 foreground classes on VOC12 and remains competitive across diverse Cityscapes scenes.
5. Conclusion
Deep layer cascade (LC) improves semantic segmentation through difficulty-aware processing, region convolution, and joint end-to-end optimization. It targets both state-of-the-art accuracy and real-time speed on PASCAL VOC and Cityscapes.
- LC trains earlier stages to handle easy, confident regions while forwarding harder regions progressively to deeper stages.
- Region convolution accelerates training and testing because each stage processes only part of the input.
- LC jointly optimizes feature learning for different regions in an end-to-end framework, achieving state-of-the-art performance on PASCAL VOC and Cityscapes.
(a) input image (b) ground truth (c) LC
Figure 9 compares an input image, its ground-truth label map, and the corresponding LC label map, with ambiguous regions marked in the ground truth.
- The input image is shown in panel (a).
- The ground-truth label map appears in panel (b), with white labels indicating ambiguous regions.
- The LC label map is shown in panel (c).