Source-linked AI summary
Improving Semantic Segmentation via Decoupled Body and Edge Supervision
Xiangtai Li, Xia Li, Li Zhang, Guangliang Cheng, Jianping Shi, Zhouchen Lin, Shaohua Tan, Yunhai Tong
TL;DR
Semantic segmentation must address both inconsistent object interiors and imprecise boundaries, while prior approaches tend to emphasize one aspect. The paper decouples body and edge features, warps features with a learned flow field, and supervises the parts separately; it reports state-of-the-art results across four road-scene datasets with lightweight modules.
Problem
Existing segmentation approaches separately target object inner consistency through global context or boundary detail through feature fusion, leaving a need to address both aspects.
Method
The framework warps features toward object interiors to generate body representations, obtains residual edge features by subtraction, and applies decoupled supervision to both.
Results
The framework achieves state-of-the-art results on Cityscapes, CamVid, KITTI, and BDD while retaining lightweight deployment across FCN architectures.
Takeaways & Limitations
Explicitly modeling body consistency and edge preservation jointly improves semantic segmentation within a unified, lightweight framework.
Takeaways & Limitations
The body-edge decomposition is motivated by an assumption that smooth interior representations and sharper boundary representations correspond to low- and high-frequency features.
Abstract
from arXiv · showhide
Existing semantic segmentation approaches either aim to improve the object's inner consistency by modeling the global context, or refine objects detail along their boundaries by multi-scale feature fusion. In this paper, a new paradigm for semantic segmentation is proposed. Our insight is that appealing performance of semantic segmentation requires \textit{explicitly} modeling the object \textit{body} and \textit{edge}, which correspond to the high and low frequency of the image. To do so, we first warp the image feature by learning a flow field to make the object part more consistent. The resulting body feature and the residual edge feature are further optimized under decoupled supervision by explicitly sampling different parts (body or edge) pixels. We show that the proposed framework with various baselines or backbone networks leads to better object inner consistency and object boundaries. Extensive experiments on four major road scene semantic segmentation benchmarks including \textit{Cityscapes}, \textit{CamVid}, \textit{KIITI} and \textit{BDD} show that our proposed approach establishes new state of the art while retaining high efficiency in inference. In particular, we achieve 83.7 mIoU \% on Cityscape with only fine-annotated data. Code and models are made available to foster any further research (\url{https://github.com/lxtGH/DecoupleSegNets}).
1 Introduction
The paper frames semantic segmentation as requiring explicit handling of both object interiors and boundaries, then proposes decoupled body-edge modeling with separate supervision.
- FCNs struggle with long-range pixel relationships and blurred fine details, causing ambiguity inside objects and imprecise boundaries.
- Prior methods separately enlarge receptive fields or inject boundary information, but they ignore interaction between object bodies and edges.
- The framework decouples segmentation features into low-frequency body and high-frequency edge representations, reflecting analogous image decomposition.
- A learned offset field warps pixels toward object interiors to generate consistent body features, while subtracting them yields edge features.
- Body features receive edge-ignored mask supervision, edge features receive edge-mask supervision, and both refined features are merged for prediction.
- The lightweight framework plugs into state-of-the-art segmentation methods with negligible cost and achieves top performance on four scene-parsing datasets.
2 Related work
Related work improves segmentation through contextual modeling, boundary processing, or multitask learning, but existing approaches have distinct limitations in boundary reliability and body-edge interaction.
- Semantic segmentation: Contextual segmentation methods use dilated backbones, pyramid pooling, non-local operators, and self-attention to capture broader pixel context.
- Boundary processing: Boundary-processing methods refine localization through structural operators, affinity modeling, random walks, or edge prediction from intermediate features.
- Boundary processing: Edge-assisted approaches can propagate errors from incorrect edge estimates, while overfitting edges may introduce noise and inferior results.
- Multi task learning: Multitask architectures learn shared representations for complementary tasks, including simultaneous segmentation and boundary detection.
3 Method
The framework refines segmentation features by separately generating object-body representations and preserving edge details, then jointly supervises the resulting components. It uses flow-based warping, residual edge extraction, specialized losses, and lightweight integration into FCN architectures.
- Decoupled feature framework: The module decomposes features into body and edge components, generating the body through context aggregation and deriving the edge by subtracting it from the input.The refined feature combines the generated body with an edge-preservation transformation of the residual feature.
- Body generation: A learned two-dimensional flow field warps pixels toward object interiors to produce more consistent body representations.Flow generation uses an encoder-decoder pipeline, upsampling, concatenation with the feature map, and a convolutional flow predictor.
- Edge preservation: The edge preservation module concatenates the residual feature with lower-level features and fuses them using a 1×1 convolution.This supplements missing fine-detail information while targeting high-frequency terms.
- Decoupled supervision: Training jointly supervises body, edge, and final segmentation outputs rather than supervising only the final map.The total loss combines body, edge, and final losses, with λ1, λ2, and λ3 set to 1 by default.
- Decoupled supervision: Body training relaxes object boundaries, while edge training uses boundary-aware binary and cross-entropy losses with online hard-example mining.The edge loss selects hard pixels using an edge threshold and balances its terms with λ4 = 25 and λ5 = 1.
- Network integration: The lightweight module can be inserted into Deeplabv3+, PSPNet, and other FCN architectures to refine feature representations.The illustrated deployment places the module after ASPP in Deeplabv3+; native FCNs append it after the backbone output layer.
4 Experiment
Experiments evaluate the framework across Cityscapes and other road-scene benchmarks, baseline architectures, ablations, boundary metrics, and visualizations. The results show gains in segmentation performance, object boundaries, and complementary body–edge representations.
- Improvements over baseline model: 3.5% and 3.0% mIoU improvements are obtained over ResNet-50 and ResNet-101 naive dilated FCN baselines, respectively.The ResNet-50-based model is also 2.2% higher than the ResNet-101 baseline.
- Ablation studies on decoupled supervisions: 1.8% improvement is achieved when all three separated supervision losses are combined, while removing edge cues and hard pixel mining causes drops of about 0.8% and 1.1%.Adding body loss contributes 0.5%, adding edge loss contributes 0.4%, and binary cross-entropy has no effect without direct segmentation supervision.
- Improvements upon different base models: Around 0.9%-1.5% mIoU improvements are reported when applying the framework to PSPNet and Deeplabv3+ with various backbones.The experiments are presented as evidence of generality across stronger base models.
- Improvement analysis: The framework improves object boundaries by a significant margin across FCN and Deeplabv3+, and is slightly better than GSCNN across four thresholds.Boundary quality is evaluated using mean F-score over 19 classes.
- Improvement analysis: Category-level gains differ by baseline: FCN improves mainly on large objects, whereas Deeplabv3+ and PSPNet improve mainly on small objects.The reported examples include large objects such as bus and car, and small objects such as traffic light and pole.
- Visualization on decoupled features and flow field: The learned flow field points inward on FCN objects but is sparse and boundary-focused for Deeplabv3+, while merged features show enhanced boundaries and complementary body and edge representations.The differing flow patterns are consistent with the reported concentration of improvements on large objects for FCN and small objects for Deeplabv3+.
5 Conclusions
The framework decouples semantic features into body and edge components, using lightweight modules and separate supervision to jointly improve object consistency and boundaries.
- The framework decouples features into body and edge parts to jointly handle inner object consistency and fine-grained boundaries.Body generation warps features toward object interiors, while edge features are obtained by subtraction.
- Decoupled loss samples pixels from different parts to supervise body and edge modules during training.
- Lightweight modules can be deployed in FCN architectures for end-to-end training.
- The approach achieves state-of-the-art results on Cityscapes, CamVid, KITTI, and BDD road scene parsing datasets.
7 More Experimental Details
Additional experiments analyze boundary quality, component design, and compatibility with related methods, while Cityscapes test results show broad category-level gains.
- Related methods are integrated by replacing the Body Generation module while retaining the supervisions and edge preservation module.
- Boundary improvements are evaluated using F-score on Deeplabv3+ across classes and four boundary thresholds, including G-SCNN for comparison.
- Component-design experiments on the Cityscapes validation set examine alternatives within the body-generation module.
- Cityscapes test results with fine-annotated data show that the method outperforms existing approaches in 18 of 19 categories and achieves 82.8% mIoU.
8 More Visualisation Results
Visualizations compare the method with FCN and Deeplabv3+, illustrate decoupled representations and flow fields, and show fine-grained mask refinements.
- Cityscapes test tables report 82.8% mIoU with fine annotations and 83.7 mIoU for the method under the stated comparison settings.
- Visual comparisons report that the method addresses inner blur in large FCN patterns and missing details or inconsistency on small Deeplabv3+ objects.
- Decoupled feature visualizations show Fbody, the residual F − Fbody, reconstructed feature F̂, and edge-prior prediction b with tb = 0.8.
- Flow-field visualizations compare fields generated from FCN and Deeplabv3+ inputs, with directions encoded by a color map and direction map.
- Fine-grained mask examples highlight refined regions in red boxes.