Source-linked AI summary
Joint Semantic Segmentation and Boundary Detection using Iterative Pyramid Contexts
Mingmin Zhen, Jinglu Wang, Lei Zhou, Shiwei Li, Tianwei Shen, Jiaxiang Shang, Tian Fang, Quan Long
TL;DR
Semantic segmentation and semantic boundary detection are tightly coupled, but prior approaches do not fully exploit their dual relationship. The paper introduces RPCNet, which iteratively exchanges pyramid context, fuses mask-derived spatial gradients, and applies boundary-consistency loss. It outperforms state-of-the-art methods on both tasks, achieving 81.8% mIoU on Cityscapes and gains of 9.9% AP and 6.8% MF(ODS) for boundary detection.
Problem
Semantic segmentation and semantic boundary detection are tightly coupled dual tasks, while boundary accuracy remains important and challenging at ambiguous image boundaries.
Method
RPCNet jointly alternates pyramid-context exchange between the tasks, fuses mask-derived spatial gradients to suppress non-semantic edges, and enforces boundary consistency with duality loss.
Results
RPCNet outperforms state-of-the-art methods on both tasks, achieving 81.8% mIoU on Cityscapes and gains of 9.9% AP and 6.8% MF(ODS) for semantic boundary detection.
Takeaways & Limitations
Jointly modeling the dual tasks with iterative context and boundary consistency improves semantic segmentation and semantic boundary detection within the reported Cityscapes evaluation.
Abstract
from arXiv · showhide
In this paper, we present a joint multi-task learning framework for semantic segmentation and boundary detection. The critical component in the framework is the iterative pyramid context module (PCM), which couples two tasks and stores the shared latent semantics to interact between the two tasks. For semantic boundary detection, we propose the novel spatial gradient fusion to suppress nonsemantic edges. As semantic boundary detection is the dual task of semantic segmentation, we introduce a loss function with boundary consistency constraint to improve the boundary pixel accuracy for semantic segmentation. Our extensive experiments demonstrate superior performance over state-of-the-art works, not only in semantic segmentation but also in semantic boundary detection. In particular, a mean IoU score of 81:8% on Cityscapes test set is achieved without using coarse data or any external data for semantic segmentation. For semantic boundary detection, we improve over previous state-of-the-art works by 9.9% in terms of AP and 6:8% in terms of MF(ODS).
1. Introduction
The paper frames semantic segmentation and semantic boundary detection as tightly coupled dual tasks and proposes a joint framework that iteratively exchanges pyramid context between them. RPCNet suppresses non-semantic edges, enforces boundary consistency, and outperforms prior work on both tasks.
- RPCNet jointly models semantic segmentation and semantic boundary detection because semantic boundaries are more tightly coupled to segmentation than binary edges.The framework iteratively correlates the two tasks through shared pyramid context.
- The iterative pyramid context module alternately uses one task's pyramid context to refine the other task's feature map.This design operationalizes the dual relationship between segmentation and boundary detection.
- Duality loss enforces consistency between the predicted mask boundary and boundary ground truth, improving boundary-pixel accuracy during end-to-end training.The loss is differentiable through pixel-wise operations.
- 81.8% mIoU is achieved on the Cityscapes test set, while semantic boundary detection improves over state of the art by 9.9% AP and 6.8% MF(ODS).Training uses only fine-annotated trainval data.
- Spatial gradient fusion combines boundaries derived from the mask probability map with semantic-boundary probabilities to suppress non-semantic edges.The derived boundary provides an initial semantic boundary estimate for cleaner results.
2. Related work
Prior work uses auxiliary shape or boundary information, but RPCNet treats semantic segmentation and boundary detection as complementary tasks that iteratively exchange context. Its design also constrains segmentation with the mask's outer contour.
- Context aggregation methods use dilated convolutions, pyramid pooling, or global pooling to strengthen feature representations.These methods motivate contextual processing but do not establish the joint task interaction described for RPCNet.
- Earlier methods use two-stream shape processing, boundary-as-class supervision, or boundary-aware feature propagation to improve segmentation representations.These approaches provide related precedents for incorporating boundary information.
- RPCNet differs from prior auxiliary-edge methods by iteratively coupling semantic segmentation and semantic boundary detection through pyramid context.The two tasks are described as complementary and more consistent than binary boundary information.
- RPCNet constrains semantic segmentation with the semantic mask's outer contour through duality loss, improving boundary accuracy of the mask.This links the boundary task directly to the segmentation output.
3. Approach
RPCNet jointly refines semantic segmentation and semantic boundary detection through iterative pyramid context interaction. It derives and fuses spatial-gradient boundaries, while adding boundary consistency to the segmentation loss.
- 3.1. Architecture: Iterative pyramid context repeatedly exchanges multi-scale context between segmentation and boundary-detection features to refine both tasks.At each level, high-level and same-level feature maps refine lower-level representations, and the two tasks interact through contextual propagation.
- 3.2. Iterative Pyramid Context Module: The PCM captures context by partitioning feature maps into patches, pooling each patch, applying corresponding convolution, and upsampling the resulting context features.The refined context is combined with the input feature map through element-wise multiplication and summation.
- 3.3. Spatial Gradient ∇M Fusion: Spatial gradient derivation obtains semantic boundaries from the segmentation mask, using adaptive average pooling with kernel size k = 3 to control boundary width.The gradient compares the mask probability map with its pooled version through an absolute difference.
- 3.3. Spatial Gradient ∇M Fusion: The inferred gradient boundary is concatenated with the learned boundary probability map and processed by grouped convolution to produce the final semantic boundary prediction.This fusion is intended to suppress non-edge pixels and localize more detailed boundaries.
- 3.4. Duality Loss: Duality loss supplements pixelwise segmentation cross-entropy with an L1 consistency term between the derived mask boundary and the ground-truth semantic boundary.The added term measures boundary consistency and is weighted by λ1 in the total segmentation loss.
4. Experiments
Experiments evaluate RPCNet on Cityscapes through component ablations, iterative-step studies, comparisons with state-of-the-art methods, and visualizations for both tasks.
- Ablation experiments: 0.62% segmentation, 1.09% MF, and 0.94% AP improvements result from PCM embedding compared with the setup without PCM.The ablation evaluates duality loss, spatial-gradient fusion, and PCM components with S fixed to 8.
- Ablation experiments: 80.43% mIoU and 75.54% / 75.14% MF / AP are obtained at S = 8, with increasing iterative steps improving both tasks.The iterative pyramid context module is evaluated from single-task training at S = 1 through S = 8, which is selected for subsequent experiments.
- Comparison with state-of-the-art works: 1.3% improvement over GSCNN is obtained on the Cityscapes validation set for semantic segmentation.GSCNN uses binary edges as a gate to boost segmentation performance.
- Comparison with state-of-the-art works: 81.8% mIoU is achieved on the Cityscapes test set without coarse-data training, while RPCNet surpasses prior methods on semantic boundary detection.The test-set comparison uses only fine-annotated trainval data; the boundary comparison reports new state-of-the-art results on MF (ODS) and AP.
- Visualization Results: Spatial-gradient fusion suppresses edge pixels that are not semantic boundaries, while duality loss produces more precise segmentation boundaries in visual comparisons.The visualizations also show recognition of a misclassified “Pole” after applying duality loss.
5. Conclusion
The paper concludes with a joint framework in which iterative multi-scale context lets semantic segmentation and boundary detection refine one another. Spatial-gradient fusion suppresses non-semantic edges, duality loss enforces boundary consistency, and Cityscapes experiments show state-of-the-art performance on both tasks.
- 5. Conclusion: Iterative pyramid context alternately refines the two tasks at multiple scales, enabling interaction between their feature maps.The framework uses context from one task to refine the other and exploits their dual relationship.
- 5. Conclusion: Spatial-gradient fusion suppresses non-semantic edge pixels, while duality loss enforces consistency between semantic-mask boundaries and boundary ground truth.These components target sparse semantic boundaries and improve semantic segmentation performance within the joint framework.
- 5. Conclusion: RPCNet outperforms current state-of-the-art works on both semantic segmentation and semantic boundary detection in comprehensive Cityscapes experiments.The conclusion reports cross-task superiority without extending the claim beyond the Cityscapes evaluation.