Source-linked AI summary
GFF: Gated Fully Fusion for Semantic Segmentation
Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, Kuiyuan Yang
TL;DR
Semantic segmentation must recover fine details from small and thin objects while using high-level semantics, but simply combining multi-level features suffers from a semantic gap. GFF addresses this with gated fully connected fusion, and the paper reports state-of-the-art results on four challenging scene-parsing datasets.
Problem
Coarse high-level feature maps lack detailed information for small and thin objects, while simply combining multi-level features suffers from a semantic gap.
Method
GFF selectively fuses multi-level feature maps through learned gates that control useful information propagation, with DFP enhancing contextual information.
Results
The method achieves state-of-the-art performance on Cityscapes, Pascal Context, COCO-stuff, and ADE20K.
Takeaways & Limitations
GFF bridges high-resolution low-semantic features and low-resolution high-semantic features while handling small and thin scene objects.
Abstract
from arXiv · showhide
Semantic segmentation generates comprehensive understanding of scenes through densely predicting the category for each pixel. High-level features from Deep Convolutional Neural Networks already demonstrate their effectiveness in semantic segmentation tasks, however the coarse resolution of high-level features often leads to inferior results for small/thin objects where detailed information is important. It is natural to consider importing low level features to compensate for the lost detailed information in high-level features.Unfortunately, simply combining multi-level features suffers from the semantic gap among them. In this paper, we propose a new architecture, named Gated Fully Fusion (GFF), to selectively fuse features from multiple levels using gates in a fully connected way. Specifically, features at each level are enhanced by higher-level features with stronger semantics and lower-level features with more details, and gates are used to control the propagation of useful information which significantly reduces the noises during fusion. We achieve the state of the art results on four challenging scene parsing datasets including Cityscapes, Pascal Context, COCO-stuff and ADE20K.
Introduction
Semantic segmentation needs features that are simultaneously high-level and high-resolution, but ConvNets distribute these properties across different levels. GFF selectively fuses multi-level features with gates, while DFP adds contextual information, achieving state-of-the-art results across four benchmarks.
- Introduction: High-level ConvNet features provide strong semantics but lose resolution, making precise prediction difficult for small objects and boundaries.The challenge includes tiny or distant objects and object boundaries.
- Introduction: Fusing high-level and low-level features can combine coarse semantic information with fine details, but their differing properties create a semantic gap.High-level maps predict large patterns coarsely, whereas low-level maps capture only limited pixels on small patterns.
- Introduction: GFF uses pixelwise gates to selectively propagate useful information among multi-level feature maps and reduce fusion noise.The gates can send useful information to other layers or receive information when the current layer is less useful.
- Introduction: DFP models contextual information after GFF by encoding and reusing context across feature levels.GFF captures detailed information in the backbone, while DFP enhances context modeling.
- Introduction: The method achieves state-of-the-art performance on Cityscapes, Pascal Context, COCO-stuff, and ADE20K.On Cityscapes, it reports 82.3% mIoU with ResNet101 and 83.3% mIoU with WiderResNet using only fine-labeled data.
Related Work
Prior segmentation methods address missing context and fine details through contextual operators or multi-level fusion. GFF differs by simultaneously fusing multi-level feature maps through gating.
- Related Work: Context modeling methods aggregate information across spatial scales, positions, or pixel-to-pixel affinities to improve scene understanding.Examples include pyramid pooling, ASPP-style context collection, and non-local operators.
- Related Work: Existing multi-level fusion methods address the top layer’s lack of fine detail, while the paper identifies contextual information as a separate challenge.The related-work discussion frames both context and detail as missing information in segmentation features.
- Related Work: GFF differs from these approaches by fusing multi-level feature maps simultaneously through a gating mechanism.The paper reports that the resulting method surpasses state-of-the-art approaches.
Method
The method selectively fuses multi-level feature maps with GFF gates, then uses DFP to add contextual information before final segmentation. Its architecture connects backbone features across levels while regulating information flow to preserve useful details and semantics.
- Multi-level Feature Fusion: Unlike concatenation, addition, and FPN’s top-down pathway, GFF fuses feature maps in a fully connected manner while suppressing redundant or useless information.Basic fusion strategies combine features without measuring their usefulness, mixing massive useless features with useful features.
- Network Architecture and Implementation: The overall network uses a backbone and PPM, applies GFF to multi-level backbone features, and passes the resulting pyramid through DFP before concatenation for segmentation.The backbone supplies features at different levels, while DFP produces final context-enriched feature maps.
- Gated Fully Fusion: GFF selectively aggregates multi-level features by measuring each feature vector’s usefulness and controlling information propagation with gates.Each level has a gate map that regulates whether information is sent to or received by another level.
- Gated Fully Fusion: A feature from level i reaches level l only when Gi(x, y) is large and Gl(x, y) is small, indicating useful information at i and missing information at l.The duplex gating mechanism regulates both sender and receiver sides and avoids redundancy by receiving information only where current features are useless.
- Dense Feature Pyramid: DFP densely connects feature pyramids so each current pyramid receives all preceding pyramid outputs and produces features used for final prediction.Its fusion function Hi is implemented by a single convolution layer, and the module adds contextual information after GFF.
- Network Architecture and Implementation: GFF visualizations compare PSPNet and GFF predictions against ground truth, highlighting refined poles, traffic lights, and object boundaries.The figure’s final column shows parts refined by GFF.
Experiment
Experiments on Cityscapes and other scene-parsing datasets show that GFF, augmented with DFP, improves segmentation performance and detail handling while learning level-specific information propagation.
- Cityscapes: 80.4% mIoU is achieved by GFF on the Cityscapes validation comparison, exceeding simpler fusion strategies and showing the value of gating.Concatenation and addition only slightly improve the PSPNet baseline, while GFF improves further; adding gating to FPN yields only slight improvement.
- Cityscapes: 0.8% mIoU is added by DFP after GFF, while multi-scale inference is evaluated as a separate improvement strategy.DFP encodes contextual information into each feature map and further improves performance in the Cityscapes ablation.
- Cityscapes: GFF achieves the highest IoU on 15 of 19 Cityscapes categories, with large improvements in small or thin categories such as poles, lights, persons, and riders.These category-level gains are consistent with the method’s stated goal of recovering fine details.
- Visualization of Gates: Gate visualizations show higher-level features serving large structures and semantics, while lower-level features contribute local details, boundaries, and semantic supplements.Ablating gates indicates that G1 and G2 mainly support boundaries, whereas G3 and G4 mainly support large patterns such as cars.
- Results on Other Datasets: The method reaches top performance on ADE20K, Pascal Context, and COCO-Stuff, including both mIoU and pixel accuracy on ADE20K.The reported results cover ResNet50 and ResNet101 on ADE20K and top performance on COCO-Stuff.
Conclusion
GFF fully fuses multi-level feature maps using learned gate maps, bridging high-resolution low-semantic features with low-resolution high-semantic features. The approach achieves state-of-the-art semantic segmentation results and handles small and thin objects.
- GFF fully fuses multi-level feature maps under the control of learned gate maps.
- The module bridges the gap between high resolution with low semantics and low resolution with high semantics.
- GFF achieves new state-of-the-art results on four challenging scene parsing datasets.
- Fusing missing low-level features into each pyramid level indicates that GFF can handle small and thin scene objects.
More visualization of Gates
The paper provides additional visualizations of learned gate maps on Cityscapes and ADE20K. These visualizations illustrate the information-regulation behavior of the gates.
- Additional learned gate-map visualizations are provided for Cityscapes and ADE20K.
- On Cityscapes, each G_i denotes the output of the ith layer’s gate and shows how gates control information propagation.
- On ADE20K, G_i represents the gate map of the ith layer.