Source-linked AI summary
Boundary-Aware Feature Propagation for Scene Segmentation
Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, Gang Wang
TL;DR
Scene segmentation must increase feature similarity within the same segment without erasing distinctions between different segments. The paper learns boundaries as an additional semantic class, uses UAGs for efficient image-wide propagation, and builds BFP to control feature exchange. It reports new state-of-the-art segmentation performance consistently on PASCAL-Context, CamVid, and Cityscapes.
Problem
Scene segmentation needs feature similarity within the same object while preserving feature discrimination between different objects.
Method
The method learns boundaries as an additional semantic class and uses UAG-structured images with BFP to guide region-aware feature propagation.
Results
The proposed BFP achieves new state-of-the-art segmentation performance consistently on PASCAL-Context, CamVid, and Cityscapes.
Takeaways & Limitations
Boundary information can guide strong within-segment and weak between-segment connections during feature propagation.
Abstract
from arXiv · showhide
In this work, we address the challenging issue of scene segmentation. To increase the feature similarity of the same object while keeping the feature discrimination of different objects, we explore to propagate information throughout the image under the control of objects' boundaries. To this end, we first propose to learn the boundary as an additional semantic class to enable the network to be aware of the boundary layout. Then, we propose unidirectional acyclic graphs (UAGs) to model the function of undirected cyclic graphs (UCGs), which structurize the image via building graphic pixel-by-pixel connections, in an efficient and effective way. Furthermore, we propose a boundary-aware feature propagation (BFP) module to harvest and propagate the local features within their regions isolated by the learned boundaries in the UAG-structured image. The proposed BFP is capable of splitting the feature propagation into a set of semantic groups via building strong connections among the same segment region but weak connections between different segment regions. Without bells and whistles, our approach achieves new state-of-the-art segmentation performance on three challenging semantic segmentation datasets, i.e., PASCAL-Context, CamVid, and Cityscapes.
1. Introduction
The paper proposes boundary-aware propagation for scene segmentation, combining learned semantic boundaries with efficient graph-based feature propagation. This design strengthens within-segment similarity while preserving discrimination across segments.
- Scene segmentation requires similar features within objects while retaining distinct features across different objects.
- Boundary-aware feature propagation builds strong same-segment connections and weak cross-segment connections to control information flow.This propagates local features within regions while using learned boundaries to regulate exchange between regions.
- The method learns object boundaries as an additional semantic class, making boundary layout available for segmentation improvement.Boundary labels are generated from existing segmentation labels and require little network modification.
- Unidirectional acyclic graphs model image-wide pixel connections while enabling faster parallel propagation than conventional graph scanning.UAGs process pixels in the same row or column in parallel and use fewer parameters than convolutional methods.
- The paper reports new state-of-the-art performance on PASCAL-Context, CamVid, and Cityscapes.
2. Related work
Prior scene-segmentation work aggregates context with convolutional, attention, and graphic models, while boundary detection has largely been treated as a separate task. This paper instead uses UAGs and learned boundary information to guide fast, region-aware propagation.
- Scene-segmentation research includes CNN-based methods such as FCN and approaches using multi-scale or contextual feature processing.
- Graphic models represent dependencies among pixels or superpixels using structures including MRFs, CRFs, Graph LSTM, UCGs, and DAGs.
- The proposed UAGs propagate local features quickly in parallel and use learned boundaries to create stronger same-segment dependencies and weaker cross-segment dependencies.
- Earlier boundary-detection methods primarily optimize boundary accuracy rather than producing semantic boundary information for high-level tasks.
3. Approach
The approach combines semantic boundary detection with efficient UAG-based propagation to spread context within segments while preserving distinctions between different segments.
- 3.1. Semantic Boundary Detection: Semantic boundary detection adds a boundary class to segmentation, making the network aware of boundary layout and improving parsing near disputed regions.The boundary task is jointly embedded with semantic object parsing, allowing segmentation to suppress non-semantic internal edges while boundary detection filters noise.
- 3.2. Unidirectional Acyclic Graphs: Four directional DAGs are decomposed into six unidirectional acyclic graphs that process rows and columns in parallel.This decomposition reduces the number of loops for each DAG from H×W to H+W, accelerating feature propagation.
- 3.3. Boundary-Aware Feature Propagation: BFP builds unimpeded connections within the same segment and controlled connections between different segments to improve feature similarity without erasing discrimination.This addresses the risk that unselective propagation assimilates features and weakens their discrimination.
- 3.3. Boundary-Aware Feature Propagation: UAG-structured features are propagated along unidirectional paths, with hidden states from four directional UAGs fused into the final output.The propagation is formulated in one dimension for clarity and extended to the image through parallel horizontal and vertical processing.
- 3.3. Boundary-Aware Feature Propagation: The boundary-aware feature propagation module converts boundary confidence into propagation confidence to control information flow across segment boundaries.Higher boundary probability produces smaller propagation probability, suppressing cross-boundary propagation while allowing stronger propagation elsewhere.
4. Experiments
The experiments evaluate implementation choices, propagation efficiency, boundary-aware ablations, boundary-focused behavior, and performance across PASCAL-Context, CamVid, and Cityscapes.
- 4.1. Implementation Details: The network uses a dilated ResNet-101 backbone with subsampling by 8, poly learning-rate scheduling, and standard flipping and resizing augmentation.The base learning rate follows a polynomial decay, while momentum and weight decay are fixed to 0.9 and 0.0001.
- 4.1. Implementation Details: Boundary-aware ground truth adds a boundary class using an 18-pixel trimap, while trimaps that are too wide can weaken propagation around small objects.Pixels within 9 pixels of a boundary receive label N+1; the experiments evaluate with mIoU.
- 4.2. Inference Speed: UAGs run faster than DAGs because they process pixels within each row or column in parallel rather than scanning pixel by pixel.The speed comparison records inference time and loop counts across input resolutions.
- 4.3. Ablation Studies: Ablations show that UAG propagation improves over dilated FCN, while boundary guidance strengthens within-segment connections and weakens cross-segment connections.The resulting propagation makes same-segment features more similar while preserving discrimination between different segments.
- 4.3. Ablation Studies: The inferred boundary maps primarily capture semantic-segment boundaries rather than object-interior edges and support boundary-aware propagation.Boundary-focused evaluation measures mIoU within bands around boundaries.
- 4.4. Comparison with the State-of-the-Art Works: BFP outperforms prior state-of-the-art methods by a large margin on PASCAL-Context and CamVid, with category-wise evaluation also reported on Cityscapes.PASCAL-Context evaluates 59 classes, CamVid evaluates 11 categories, and Cityscapes evaluates 19 classes using fine annotations.
5. Conclusion
The paper combines semantic boundary learning, efficient UAG-based propagation, and BFP to improve feature organization for scene segmentation. It reports consistent new state-of-the-art performance on PASCAL-Context, CamVid, and Cityscapes.
- 5. Conclusion: The approach learns boundaries as an additional semantic class and uses them to control feature propagation across UAG-structured images.UAGs model undirected cyclic graph functions efficiently while aggregating holistic context.
- 5. Conclusion: BFP strengthens feature similarity within segment regions while preserving feature discrimination between different segments.The module harvests and propagates local features under inferred boundary control.
- 5. Conclusion: The method achieves consistent new state-of-the-art segmentation performance on PASCAL-Context, CamVid, and Cityscapes.The conclusion presents this result across all three named datasets.