Source-linked AI summary
Semantic Flow for Fast and Accurate Scene Parsing
Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, Yunhai Tong
TL;DR
Fast scene parsing must combine high-resolution spatial detail with strong semantics, while atrous convolutions and conventional feature-pyramid fusion have efficiency or effectiveness limitations. The paper introduces FAM, which learns Semantic Flow to align adjacent-level feature maps and propagate high-level features efficiently. SFNet achieves strong results across multiple datasets, including 80.4% mIoU at 26 FPS on Cityscapes with ResNet-18.
Problem
Fast and accurate scene parsing requires high-resolution features with strong semantics, but atrous convolutions are computationally intensive and feature-pyramid fusion can be ineffective.
Method
FAM learns Semantic Flow between adjacent feature levels and uses it to align and broadcast high-level features within a feature pyramid.
Results
The method achieves state-of-the-art results on Cityscapes, Pascal Context, and CamVid, a considerable gain on ADE20K, and 80.4% mIoU at 26 FPS on Cityscapes with ResNet-18.
Takeaways & Limitations
SFNet provides a lightweight feature-alignment strategy that trades off semantic segmentation accuracy and running-time efficiency effectively across several challenging datasets.
Abstract
from arXiv · showhide
In this paper, we focus on designing effective method for fast and accurate scene parsing. A common practice to improve the performance is to attain high resolution feature maps with strong semantic representation. Two strategies are widely used -- atrous convolutions and feature pyramid fusion, are either computation intensive or ineffective. Inspired by the Optical Flow for motion alignment between adjacent video frames, we propose a Flow Alignment Module (FAM) to learn Semantic Flow between feature maps of adjacent levels, and broadcast high-level features to high resolution features effectively and efficiently. Furthermore, integrating our module to a common feature pyramid structure exhibits superior performance over other real-time methods even on light-weight backbone networks, such as ResNet-18. Extensive experiments are conducted on several challenging datasets, including Cityscapes, PASCAL Context, ADE20K and CamVid. Especially, our network is the first to achieve 80.4\% mIoU on Cityscapes with a frame rate of 26 FPS. The code is available at \url{https://github.com/lxtGH/SFSegNets}.
1 Introduction
Scene parsing requires both detailed spatial information and strong semantic representation, but common approaches can be computationally expensive or insufficiently accurate. The paper introduces Semantic Flow and FAM to align multi-level features efficiently, forming SFNet for fast, accurate parsing.
- Motivation: Scene parsing classifies each image pixel, requiring both detailed object boundaries and strong semantic representation.Down-sampling can remove boundary information, motivating methods that preserve high-resolution features while retaining semantics.
- Motivation: Atrous convolutions preserve higher-resolution semantic features but can impose substantial computational costs.For ResNet-18, the cited frame rate drops from 57.2 FPS to 8.7 FPS after applying atrous convolutions.
- Motivation: Feature pyramid methods fuse deep semantic features with shallow high-resolution features, yet their accuracy remains unsatisfactory compared with stronger networks.The top-down fusion strategy strengthens shallow features, but the supplied passage identifies an accuracy gap.
- Proposed approach: Semantic Flow represents the relationship between feature maps at different resolutions as pixel-wise motion for dynamic alignment.It is inspired by optical flow but operates on feature maps from different network levels within the same image.
- Results: 80.4% mIoU at 26 FPS is achieved on the Cityscapes test server with ResNet-18.The paper reports state-of-the-art results across Cityscapes, Pascal Context, and CamVid, with a considerable gain on ADE20K.
- Proposed approach: FAM learns Semantic Flow between adjacent feature levels and broadcasts high-level features to high-resolution features efficiently within a feature pyramid.SFNet inserts FAM into the pyramid and is designed to improve accuracy while remaining lightweight.
2 Related Work
Scene parsing methods either preserve high-resolution features in the main pathway or distribute spatial and semantic information across network components before merging them. Related fast methods use multi-level fusion, cascades, lightweight backbones, or searched architectures, while this work visualizes semantic flow for feature alignment.
- Paradigms: Scene parsing approaches follow two paradigms: retaining spatial and semantic information together or distributing them across network components before fusion.The distinction concerns where spatial and semantic information are represented and how they are combined.
- High-resolution pathways: The first paradigm uses high-resolution feature maps and contextual modules such as pyramid pooling, atrous spatial pyramid pooling, non-local operators, and self-attention.These methods preserve resolution in later stages while modeling contextual information.
- Efficient fusion: The second paradigm fuses multi-level features or uses cascaded architectures to combine high-level semantics with high-resolution spatial information efficiently.Examples include ICNet, DFANet, and SwiftNet.
- Lightweight networks: Lightweight scene-parsing methods reduce computation through decomposed convolutions, separate spatial and semantic paths, or AutoML-based architecture search.The paper describes its method as complementary to some of these approaches.
- Semantic Flow visualization: Semantic Flow is visualized alongside feature maps, with flow orientation encoded by hue and magnitude by saturation.Feature maps are visualized by channel averaging, with larger values shown using hotter colors.
3 Method
The method builds an FPN-based scene-parsing network that uses Flow Alignment Modules to align adjacent-resolution features before fusing them. FAM predicts Semantic Flow from neighboring feature maps and warps high-level features into higher-resolution representations with lightweight operations.
- Flow Alignment Module: FAM predicts Semantic Flow between adjacent-resolution feature maps and uses it to warp top-level features for alignment.The module is designed to resolve misalignment while transferring semantic information from deeper to shallower layers.
- Flow Alignment Module: FAM combines adjacent feature maps, predicts offsets with convolutional layers, and maps each lower-level position to an upper-level point.The predicted flow field is used to define the sampling locations for the aligned feature map.
- Flow Alignment Module: Bilinear sampling interpolates four neighboring values around each warped point to produce the final aligned feature.The sampling operation is differentiable and uses distance-based bilinear weights.
- Flow Alignment Module: Unlike deformable convolution, FAM uses both higher- and lower-level features to align feature-map positions rather than adapt kernel locations.The paper distinguishes FAM’s feature-alignment role from deformable convolution’s attention-like behavior.
- Flow Alignment Module: FAM is lightweight and end-to-end trainable, using one 3×3 convolution and one parameter-free warping operation with minor extra computation overhead.The module can be inserted multiple times into a network.
- Network Architecture: SFNet uses a bottom-up backbone encoder and an FPN-style top-down decoder equipped with multiple FAMs.The architecture compares several ImageNet-pretrained backbone families and replaces some upsampling operations with FAM-based alignment.
4 Experiments
Experiments evaluate SFNet’s accuracy, efficiency, architecture, and FAM design across Cityscapes, PASCAL Context, ADE20K, and CamVid. Results show improved segmentation with lightweight computation and strong speed–accuracy trade-offs.
- Architecture ablations: FAM improves mIoU across heavy and lightweight backbones with only slightly extra computational cost.The ablation uses an FPN decoder with a PPM head as a strong baseline.
- Real-time comparisons: On Cityscapes, single-scale comparisons report 74.5% mIoU for DF1, 60 FPS with DF2, and 78.9% mIoU with ResNet-18.Input sizes are considered for fair real-time comparison, and speeds are tested at full image resolution.
- Cityscapes results: 80.4% mIoU at 26 FPS establishes SFNet’s Cityscapes accuracy–speed trade-off result.The result uses multi-scale with flip testing and is reported as a state-of-the-art trade-off.
- Cityscapes results: 1.1% higher mIoU than PSPNet with 11% of its computation is achieved by the ResNet-18 SFNet model.The comparison is reported on the Cityscapes test set.
- Cross-dataset results: On CamVid, FAM improves the DF2 baseline by 3.2% mIoU, while ResNet-18 reaches 73.8% mIoU at 35.5 FPS.CamVid evaluation uses 960 × 720 input images with single-scale inference.
5 Conclusion
The conclusion presents Semantic Flow alignment as a way to fuse high-level features into high-resolution feature maps while avoiding atrous convolutions. The resulting network targets a balance between semantic strength, spatial detail, accuracy, and efficiency.
- 5 Conclusion: SFNet aligns multi-level feature maps so high-level features are fused into low-level, high-resolution representations.The design discards atrous convolutions to reduce computation overhead.
- 5 Conclusion: The proposed flow alignment module is intended to provide a trade-off between semantic representation, spatial detail, accuracy, and computational efficiency.The conclusion frames this balance as the network’s central objective.
6 Supplemental Parts
The supplemental material covers additional Cityscapes and other-dataset details and states that the codebase will be open-sourced.
- The supplement contains more details on Cityscapes and detailed settings for other datasets.
- The authors state that they will open-source the codebase.
7 Supplemental Experiments on Cityscapes
Supplemental Cityscapes experiments examine category-level gains, feature alignment visualizations, learned semantic flows, training with Mapillary Vistas, and detailed efficiency comparisons.
- Detailed improvement on baseline models: More than 19% mIoU improvement is reported for the truck category over the FPN decoder with PPM head baseline.The comparison uses a ResNet-101 backbone on the Cityscapes validation set.
- More structured feature visualization on FAM: FAM aligns last-stage and next-to-last-stage outputs into more structured representations with clearer shapes and more accurate boundaries.
- More Visualization of Learned Flow: Figure 8 visualizes semantic flow fields from three FAMs in ascending resolution order during decoding.
- More training details using Mapillary Vistas: SFNet is pretrained on Mapillary Vistas using both training and validation images before fine-tuning on Cityscapes fine-annotated data.The procedure uses 50,000 iterations for pretraining and 50,000 iterations for Cityscapes fine-tuning.
- Detailed state-of-the-art model comparison: SFNet achieves the best state-of-the-art comparison performance with fewer GFlops under seven-scale, flipped inference.
- Detailed improvement on baseline models: Table 7 reports that FAM boosts performance for almost all categories on the Cityscapes validation set.
8 Detailed Experiment Settings on Other Datasets:
The supplemental settings describe evaluation and dataset details for PASCAL Context, ADE20K, CamVid, and Cityscapes test-set comparison results.
- PASCAL Context: PASCAL Context contains 4,998 training images and 5,105 validation images, with multi-scale evaluation and horizontal flipping.
- ADE20K: ADE20K contains 20K training and 2K validation images across 150 semantic classes, evaluated with multi-scale testing and horizontal flipping.
- Cityscapes test set: SFNet achieves 81.8% mIoU on the Cityscapes test set with much less GFlops than compared models.
- CamVid: CamVid provides pixel-wise annotations for 11 categories across 367 training, 101 validation, and 233 testing images, using single-scale final testing.