Source-linked AI summary
Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
Towaki Takikawa, David Acuna, Varun Jampani, Sanja Fidler
TL;DR
Semantic segmentation networks often process color, shape, and texture together despite their different roles. Gated-SCNN separates shape into a parallel stream, uses classical-stream activations to gate it, and reports sharper boundaries and stronger Cityscapes performance, particularly for thin and small objects.
Problem
Dense CNN segmentation architectures process color, shape, and texture together even though these information types differ in relevance for recognition.
Method
Gated-SCNN uses a classical segmentation stream and a separate shape stream connected by gates, with boundary supervision and late feature fusion.
Results
Gated-SCNN achieves state-of-the-art Cityscapes results, improving over DeepLab-v3+ by more than 1.5% mIoU and 4% F-boundary score, with up to 7% IoU gains for thin and small objects.
Takeaways & Limitations
The architecture produces sharper predictions around object boundaries and significantly boosts performance on thinner and smaller objects.
Takeaways & Limitations
The experiments do not use Cityscapes’ coarse annotations because they are not ideal for supervising the shape stream with boundary ground truth.
Abstract
from arXiv · showhide
Current state-of-the-art methods for image segmentation form a dense image representation where the color, shape and texture information are all processed together inside a deep CNN. This however may not be ideal as they contain very different type of information relevant for recognition. Here, we propose a new two-stream CNN architecture for semantic segmentation that explicitly wires shape information as a separate processing branch, i.e. shape stream, that processes information in parallel to the classical stream. Key to this architecture is a new type of gates that connect the intermediate layers of the two streams. Specifically, we use the higher-level activations in the classical stream to gate the lower-level activations in the shape stream, effectively removing noise and helping the shape stream to only focus on processing the relevant boundary-related information. This enables us to use a very shallow architecture for the shape stream that operates on the image-level resolution. Our experiments show that this leads to a highly effective architecture that produces sharper predictions around object boundaries and significantly boosts performance on thinner and smaller objects. Our method achieves state-of-the-art performance on the Cityscapes benchmark, in terms of both mask (mIoU) and boundary (F-score) quality, improving by 2% and 4% over strong baselines.
1. Introduction
Gated-SCNN separates shape processing from the classical segmentation stream and uses gating and boundary-aware supervision to improve boundary quality and performance, especially for small objects.
- Boundary supervision: A boundary-alignment loss encourages predicted segmentation masks to align with ground-truth semantic boundaries.The loss is designed to improve the quality of predicted object boundaries.
- Motivation and contribution: Gated-SCNN separates shape information into a parallel processing stream rather than combining color, shape, and texture within one deep CNN.The classical stream and shape stream remain separate until the top layers.
- Architecture: Higher-level classical-stream activations gate early shape-stream activations, filtering irrelevant information so a shallow, full-resolution shape stream can focus on boundaries.The gates denoise shape activations, while a semantic boundary loss provides local supervision.
- Results: More than 1.5% mIoU and 4% F-boundary improvements over DeepLab-v3+ were reported on Cityscapes.The evaluation explored ResNet-50, ResNet-101, and WideResNet backbones and reported improvements across them.
- Results: Up to 7% IoU improvement was obtained for thinner and smaller objects, including poles, traffic lights, and traffic signs.The method also achieved up to 6% mIoU improvement for objects at the largest evaluated distance.
2. Related Work
Prior segmentation work improved dense prediction through fully convolutional conversion, structured prediction, multiscale context, and alternative two-stream or multitask designs. Gated-SCNN instead specializes its second stream for shape-related information and uses structured interaction between segmentation and boundary tasks.
- Semantic Segmentation: Earlier methods converted image-classification networks into fully convolutional networks for end-to-end semantic segmentation.Other approaches added structured prediction modules such as CRFs to improve segmentation, particularly near object boundaries.
- Semantic Segmentation: PSPNet and DeepLab improved performance and inference speed with feature-pyramid pooling that aggregates multiscale context.Their pooling modules combine features across multiple scales.
- Semantic Segmentation: A related two-stream network used its second stream to recover high-resolution features lost through pooling, whereas Gated-SCNN specializes that stream for shape information.The distinction is the functional role assigned to the second branch.
- Multitask Learning: Multitask architectures learn shared representations with multiple task losses, while Gated-SCNN uses task duality to enforce a structured representation rather than train a multitask network.The paired tasks are semantic segmentation and semantic boundary prediction.
- Gated Convolutions: Prior gated-convolution methods applied gating to temporal convolutions, image inpainting, and conditional image generation; Gated-SCNN applies gated convolutions to shape processing in segmentation.Its shape stream is constrained to boundary-related information before fusion with regular-stream features.
3. Gated Shape CNN
GSCNN separates semantic-region processing from boundary-focused shape processing, using gated interactions and fusion to refine segmentation around object boundaries. Joint boundary and semantic supervision, plus dual-task regularization, supports this design.
- Architecture: GSCNN uses a regular segmentation stream and a separate shape stream followed by a fusion module.The regular stream can use a standard backbone, while the shape stream processes boundary-related information.
- Shape Stream: The shape stream processes semantic boundaries from image gradients and regular-stream features using residual blocks, gated convolution layers, and local supervision.Ground-truth binary edges supervise the predicted boundary map with binary cross-entropy.
- Fusion Module: The fusion module combines regular-stream region features with shape-stream boundary features to produce refined K-class pixel probabilities while preserving multi-scale context.The architecture uses an Atrous Spatial Pyramid Pooling module for multi-scale fusion.
- Gated Convolutional Layer: Gated convolution layers use higher-level regular-stream information to deactivate irrelevant shape-stream activations without directly incorporating regular-stream features.This enables a shallow shape stream operating at high image resolution.
- Training: GSCNN jointly trains semantic segmentation and boundary prediction with BCE and CE losses, while a dual-task regularizer enforces consistency in boundary space.The regularizer emphasizes pixels that are boundaries in both predicted and ground-truth boundary representations.
4. Experimental Results
Experiments on Cityscapes evaluate GSCNN across backbones, metrics, distances, components, and qualitative examples. The method improves region and boundary accuracy, especially for smaller and thinner objects, while its gates emphasize boundary information.
- Experimental setup: Cityscapes experiments evaluate GSCNN across multiple backbone architectures, quantitative metrics, distance settings, ablations, and qualitative results.The study uses ResNet-50, ResNet-101, and Wide-ResNet regular streams and evaluates region, boundary, and distance-based performance.
- Quantitative evaluation: 2% improvement in mIoU is reported on the Cityscapes validation set, with notable gains for motorcycles, traffic signs, traffic lights, and poles.The comparison measures region accuracy on full images against the baselines.
- Quantitative evaluation: Nearly 4% higher boundary F-score is achieved than the baseline in the strictest evaluation regime.The comparison reports models trained on Cityscapes fine annotations and uses single-scale inference.
- Distance-based evaluation: The performance gap over DeepLabV3+ grows from 2% without cropping to nearly 6% at crop factor 400, indicating stronger results for distant small objects.The distance proxy uses crops around an approximate vanishing point.
- Cityscapes benchmark: On the Cityscapes test set, GSCNN consistently outperforms strong published baselines and ranks first among published methods that do not use coarse data.Some competing methods use extra coarse training data, whereas this model does not.
- Ablation: Ablations report 1–2% mIoU and around 3% boundary-alignment improvements, while Dual Task loss provides up to 3% improvement in boundary F-score.The ablations compare the shape stream, GCL, image gradients, and Dual Task loss under different regular-stream backbones and thresholds.
- Qualitative results: The gates emphasize low-level edges in early stages and object-level boundaries later, producing class-agnostic boundary maps that are fed to fusion.Qualitative results show sharper boundaries, including preserved structure for traffic lights and improved delineation of poles and other thin objects.
5. Conclusion
GSCNN separates shape processing into a parallel stream and connects it to the regular stream through gating and a dual-task loss. The resulting architecture produces sharper object boundaries and improves performance on thinner and smaller objects on Cityscapes.
- Contribution: GSCNN wires shape information into a separate parallel stream within a two-stream semantic-segmentation CNN.The architecture uses a classical stream alongside a dedicated shape stream.
- Contribution: New gates connect intermediate layers, while a dual-task loss exploits the relationship between semantic segmentation and boundary prediction.The method uses these components to structure information flow between segmentation and boundary processing.
- Conclusion: The architecture produces sharper predictions around object boundaries and significantly boosts performance on thinner and smaller objects.The conclusion reports state-of-the-art results on the challenging Cityscapes dataset over strong baselines.