Source-linked AI summary
Semantic Correlation Promoted Shape-Variant Context for Segmentation
Henghui Ding, Xudong Jiang, Bing Shuai, Ai Qun Liu, Gang Wang
TL;DR
Fixed spatial regions are poorly matched to the varying shapes and scales of contextual information needed for semantic segmentation. The paper learns semantic-correlation masks with paired convolution, applies shape-variant convolution and labeling denoising, and reports consistent state-of-the-art performance across six public datasets.
Problem
Diverse object shapes and layouts make predefined fixed context regions ineffective or inefficient for aggregating useful semantic information.
Method
The network infers semantic-correlation masks with paired convolution, uses shape-variant convolution for adaptive context aggregation, and denoises low-level features with higher-level features.
Results
The proposed segmentation network achieves new state-of-the-art performance consistently on six public semantic segmentation datasets.
Takeaways & Limitations
Context aggregation is guided by semantic correlation, allowing each pixel to use a diverse shape and scale of context rather than a fixed spatial region.
Abstract
from arXiv · showhide
Context is essential for semantic segmentation. Due to the diverse shapes of objects and their complex layout in various scene images, the spatial scales and shapes of contexts for different objects have very large variation. It is thus ineffective or inefficient to aggregate various context information from a predefined fixed region. In this work, we propose to generate a scale- and shape-variant semantic mask for each pixel to confine its contextual region. To this end, we first propose a novel paired convolution to infer the semantic correlation of the pair and based on that to generate a shape mask. Using the inferred spatial scope of the contextual region, we propose a shape-variant convolution, of which the receptive field is controlled by the shape mask that varies with the appearance of input. In this way, the proposed network aggregates the context information of a pixel from its semantic-correlated region instead of a predefined fixed region. Furthermore, this work also proposes a labeling denoising model to reduce wrong predictions caused by the noisy low-level features. Without bells and whistles, the proposed segmentation network achieves new state-of-the-arts consistently on the six public segmentation datasets.
1. Introduction
The paper argues that fixed spatial context is poorly suited to objects with diverse shapes and layouts. It proposes semantic-correlation-guided, shape-variant context aggregation alongside denoising of noisy low-level features.
- Semantic segmentation classifies every image pixel into predefined categories and benefits from surrounding context for object discrimination.
- Fixed context regions can overlap unrelated objects, weaken location and shape information, and include irrelevant pixels for differently shaped objects.
- A paired convolution and Gaussian mapping infer pixelwise semantic correlation to generate a semantic-correlated shape mask.
- Shape-variant convolution uses location-invariant parameters and location-variant masks to aggregate context from semantic-dependent regions.
- The labeling denoising model uses higher-level features to attenuate noise in low-level features before aggregation, reducing errors caused by noisy low-level information.
- The paper reports new state-of-the-art performance consistently on six public semantic segmentation datasets.
2. Related work
Prior work commonly aggregates context through fixed spatial windows or global representations. This paper instead uses semantic correlation to adapt context shape and suppress irrelevant features while addressing label confusion through related segmentation strategies.
- Existing context methods aggregate fixed-size rectangular or global context, which can weaken location identity and fail to represent diverse object shapes and scales.
- The proposed approach aggregates context from semantically closer regions while suppressing irrelevant information even when it is spatially close.
- Its shape-adaptive convolution learns context shapes determined by object shape, scale, and surrounding support.
- Figure 2 illustrates semantic-correlation-dependent context by boosting correlated features and suppressing others.
- Related segmentation work addresses label variety through confusion modeling, label refinement, and networks designed for confusing classes.
3. The Proposed Approach
The proposed approach learns semantic-correlated, scale- and shape-variant context regions for each pixel, replacing fixed rectangular aggregation with input-dependent receptive fields. It also uses labeling denoising to reduce errors associated with noisy low-level features.
- Motivation: Fixed rectangular context can weaken location identity and include irrelevant information because objects have diverse shapes, scales, and layouts.Pixels from different objects may require different contextual regions, while predefined receptive fields can overlap inappropriately.
- Shape-Variant Context: The semantic mask boosts information from correlated regions and suppresses irrelevant information, enabling multi-shape and multi-scale context modeling within a single layer.This design preserves shape and location identity while selecting context according to input-feature appearance.
- Semantic Correlation: The paired convolution learns pixel-to-pixel semantic correlation and maps it to a shape mask indicating each pixel’s correlated contextual region.The Gaussian mapping assigns higher semantic-correlation values when paired-convolution output discrepancy is smaller.
- Shape-Variant Context: Shape-variant convolution uses the inferred mask to weight normal learnable filters, producing location-dependent receptive fields with different effective shapes and scales.The mask transforms a fixed K×K filter into shape-variant filters for different spatial positions.
- Labeling Denoising: The labeling denoising model uses higher-level predictions to attenuate noise from low-level spatial features during decoding.The denoising component addresses incorrect labeling associated with noisy low-level information.
4. Experiments
Experiments evaluate SVCNet through training details, ablations, mask visualizations, and comparisons across six public segmentation benchmarks. Results show benefits from labeling denoising, semantic shape-variant context, and semantic-correlated masks.
- Experimental Setup: SVCNet is evaluated on six benchmarks using ResNet-101 with FCN-4s, end-to-end SGD training, and pixel accuracy, mean accuracy, and mean IoU.The benchmarks are COCO-Stuff, SIFT-Flow, CamVid, PASCAL-Person-Part, PASCAL-Context, and Cityscapes.
- Ablation Study: SVC significantly outperforms shape-fixed context across kernel sizes, while performance first improves and then declines when kernels become excessively large.The decline is attributed to losing locality information, and the gain is not explained simply by increasing parameters.
- Visualization of the Semantic Shape Mask: Visualized masks assign higher values to semantically correlated regions, including cars and roads, cows and grass, and trains and railway tracks.For the cow example, distant cows and grass receive higher values while intervening road pixels receive low values.
- Comparison with the State-of-the-Arts: Across COCO-Stuff, SIFT-Flow, CamVid, PASCAL-Person-Part, and PASCAL-Context, SVCNet outperforms previous state-of-the-art methods across metrics or by a large margin.The reported comparisons include all evaluation metrics on COCO-Stuff and SIFT-Flow, a large margin on CamVid and PASCAL-Person-Part, and a large margin on PASCAL-Context.
5. Conclusion
The method aggregates context according to semantic correlation rather than a fixed spatial window, while achieving state-of-the-art results across six public semantic segmentation datasets.
- Semantic correlation guides context aggregation, enhancing relevant information even at far locations and suppressing irrelevant information nearby.This replaces a predefined spatial-dependent window with semantic-dependent context selection.
- A semantic shape mask specifies diverse receptive-field scales and shapes for shape-variant convolution at different image positions.The masks are generated from feature semantic correlation using paired convolution.
- The proposed network achieves new state-of-the-arts consistently on COCO-Stuff, SIFT-Flow, CamVid, PASCAL-Person-Part, PASCAL-Context, and Cityscapes.