Source-linked AI summary
Learning a Discriminative Feature Network for Semantic Segmentation
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang
TL;DR
Semantic segmentation methods often produce inconsistent labels within a class and confuse adjacent classes with similar appearances. DFN combines Smooth and Border Networks to select discriminative features and sharpen semantic boundaries, achieving strong benchmark performance on PASCAL VOC 2012 and Cityscapes.
Problem
Semantic segmentation features remain insufficiently discriminative for differently appearing patches within one class and similarly appearing adjacent patches from different classes.
Method
DFN combines a Smooth Network using global context and channel attention with a Border Network using semantic boundary supervision.
Results
The proposed approach achieves 86.2% performance on the PASCAL VOC 2012 test set without MS-COCO fine-tuning and improves the base network from 72.86% to 76.65% with its Refinement Residual Block.
Takeaways & Limitations
DFN addresses intra-class consistency and inter-class distinction through a bidirectional stage-wise mechanism across Smooth and Border Networks.
Abstract
from arXiv · showhide
Most existing methods of semantic segmentation still suffer from two aspects of challenges: intra-class inconsistency and inter-class indistinction. To tackle these two problems, we propose a Discriminative Feature Network (DFN), which contains two sub-networks: Smooth Network and Border Network. Specifically, to handle the intra-class inconsistency problem, we specially design a Smooth Network with Channel Attention Block and global average pooling to select the more discriminative features. Furthermore, we propose a Border Network to make the bilateral features of boundary distinguishable with deep semantic boundary supervision. Based on our proposed DFN, we achieve state-of-the-art performance 86.2% mean IOU on PASCAL VOC 2012 and 80.3% mean IOU on Cityscapes dataset.
1. Introduction
Semantic segmentation remains challenged by features that confuse different appearances within a class and similar appearances across adjacent classes. DFN addresses these issues through Smooth and Border Networks designed to improve feature discrimination.
- Motivation: FCN-based features struggle with intra-class inconsistency across differently appearing patches and inter-class indistinction between adjacent, similarly appearing patches.The paper illustrates these problems with a cow patch mistaken for a horse and a computer case confused with a monitor.
- Motivation: DFN reframes segmentation as assigning a consistent label to a category of things rather than to each individual pixel.This macroscopic view is intended to account for both intra-class consistency and inter-class distinction.
- Smooth Network: The Smooth Network targets intra-class inconsistency with multi-scale and global context, global average pooling, and a Channel Attention Block that selects effective features.Its U-shape structure captures multi-scale context, while high-level features guide low-level feature selection stage by stage.
- Border Network: The Border Network targets inter-class indistinction by using semantic boundary supervision to enlarge feature variation on opposite sides of boundaries.This explicitly models semantic boundaries that dense recognition approaches may overlook.
- Contributions: DFN combines Smooth and Border Networks to address intra-class consistency and inter-class variation, with experiments on PASCAL VOC 2012 and Cityscapes validating the approach.The paper also describes a bottom-up Border Network with deep supervision for refining semantic boundaries.
2. Related Work
Related work spans FCN-derived segmentation architectures, global-context modules, attention mechanisms, and semantic boundary detection. DFN builds on these directions while using channel attention and boundary supervision to pursue more discriminative features.
- Encoder-Decoder: FCN-based methods commonly refine predictions by recovering spatial information lost through pooling or strided convolution.Examples include SegNet’s saved pooling indices and U-Net’s skip connections.
- Global Context: Global average pooling and its pyramid extensions have improved semantic segmentation performance across different benchmarks.The passage cites ParseNet, PSPNet, and DeepLab v3 as successive uses or extensions of global context.
- Attention Module: Attention modules focus neural networks on relevant information, while this work applies channel attention to select features across scales.The stated design is similar to the channel-selection strategy in SENet.
- Semantic Boundary Detection: Prior boundary-detection methods often concatenate feature levels, whereas DFN uses a bottom-up structure with accurate boundary supervision to optimize features at each stage.The goal is to obtain features with stronger inter-class distinction.
3. Method
The Discriminative Feature Network combines a Smooth Network for intra-class consistency with a Border Network for inter-class distinction. The Smooth Network selects discriminative multi-scale features, while the Border Network uses semantic boundary supervision to distinguish adjacent classes.
- Smooth Network: The Smooth Network combines multi-scale context, global average pooling, and channel attention to select discriminative features.Channel attention re-weights low-stage feature maps using information from adjacent stages.
- Smooth Network: Higher stages provide stronger semantic consistency, while lower stages retain finer spatial information; the Smooth Network uses the former to guide the latter.This combines accurate semantic predictions from high stages with more precise spatial predictions from low stages.
- Smooth Network: The Channel Attention Block changes feature weights because different channels and stages have different discriminative value.Its weighting applies to feature maps to enhance intra-class-consistent predictions.
- Border Network: The Border Network learns semantic boundaries with explicit supervision so features on opposite sides become more distinguishable.A bottom-up design combines detailed low-stage edge information with high-stage semantic information, using boundaries derived from segmentation ground truth.
- The DFN comprises Smooth and Border Networks targeting intra-class consistency and inter-class distinction, respectively.Its overall architecture is presented as an encoder-decoder network.
4. Experimental Results
Experiments decompose DFN into Smooth and Border Networks, evaluate their components, and report results on PASCAL VOC 2012 and Cityscapes. The components improve consistency, boundary refinement, and final segmentation performance under the reported evaluation settings.
- Experimental setup: The experiments evaluate DFN components on PASCAL VOC 2012 and report comparison results on both PASCAL VOC 2012 and Cityscapes.The study evaluates each component before presenting comparisons with other methods.
- Experimental setup: The base model uses ResNet-101 with FCN4 as the segmentation framework, trained using mini-batch SGD and multi-scale augmentation.The implementation uses five input scales and random horizontal flips.
- Smooth Network: 76.65% performance follows extending the base network to FCN4 with Refinement Residual Blocks, compared with 72.86% for the base network.These values are reported in the Smooth Network ablation sequence.
- Smooth Network: 78.20% performance follows adding global average pooling to the 76.65% configuration, while deep supervision further improves performance by almost 0.4%.Global pooling provides consistency guidance, and deep supervision refines hierarchical features.
- DFN analysis: The highest performance among tested loss-balance values occurs at λ = 0.1, while bidirectional stage-wise refinement progressively improves segmentation and boundary predictions.Smooth Network refinement proceeds top-down, whereas Border Network refinement proceeds bottom-up.
- Final evaluation: The approach is also evaluated on Cityscapes, a high-resolution urban street-scene dataset with 19 evaluated classes and fine annotations.The dataset contains 2,979 training, 500 validation, and 1,525 test images with fine annotations.
5. Conclusion
The paper reframes semantic segmentation as assigning consistent labels to object categories and proposes DFN to address intra-class consistency and inter-class distinction. DFN combines Smooth and Border Networks, improving performance on PASCAL VOC 2012 and Cityscapes.
- DFN reframes semantic segmentation as assigning a consistent semantic label to one category of objects rather than to each pixel.
- DFN contains Smooth Network and Border Network sub-networks to consider intra-class consistency and inter-class distinction.
- The authors report that DFN significantly improves performance on the PASCAL VOC 2012 and Cityscapes benchmarks.