Source-linked AI summary
CASENet: Deep Category-Aware Semantic Edge Detection
Zhiding Yu, Chen Feng, Ming-Yu Liu, Srikumar Ramalingam
TL;DR
Category-aware semantic edge detection is challenging because an edge pixel may belong to multiple semantic classes. CASENet addresses this with an end-to-end ResNet-based nested architecture and multi-label loss, improving over existing architectures and prior state-of-the-art methods on SBD and Cityscapes.
Problem
Category-aware semantic edge detection requires assigning multiple possible semantic classes to edge pixels, but related approaches have treated the task as multi-class learning.
Method
CASENet combines a nested ResNet architecture, shared bottom features for top category classifications, and a multi-label loss supervising category-wise edge outputs.
Results
CASENet improves over existing architectures and outperforms previous state-of-the-art semantic edge methods by significant margins on SBD and Cityscapes.
Takeaways & Limitations
The results support multi-label learning and the CASENet architecture for category-aware semantic edge detection.
Takeaways & Limitations
Early network layers have limited receptive fields, making semantic classification at that stage potentially unreasonable because context is important.
Abstract
from arXiv · showhide
Boundary and edge cues are highly beneficial in improving a wide variety of vision tasks such as semantic segmentation, object recognition, stereo, and object proposal generation. Recently, the problem of edge detection has been revisited and significant progress has been made with deep learning. While classical edge detection is a challenging binary problem in itself, the category-aware semantic edge detection by nature is an even more challenging multi-label problem. We model the problem such that each edge pixel can be associated with more than one class as they appear in contours or junctions belonging to two or more semantic classes. To this end, we propose a novel end-to-end deep semantic edge learning architecture based on ResNet and a new skip-layer architecture where category-wise edge activations at the top convolution layer share and are fused with the same set of bottom layer features. We then propose a multi-label loss function to supervise the fused activations. We show that our proposed architecture benefits this problem with better performance, and we outperform the current state-of-the-art semantic edge detection methods by a large margin on standard data sets such as SBD and Cityscapes.
1. Introduction
CASENet addresses category-aware semantic edge detection as a multi-label problem, allowing edge pixels to associate with multiple semantic categories. The proposed architecture and loss improve semantic edge learning and outperform prior methods on SBD and Cityscapes.
- 1. Introduction: Semantic edge detection assigns one or more object categories to each detected edge pixel, including multiple categories at contours and junctions.The formulation represents each edge pixel through category associations rather than a single class.
- 1. Introduction: CASENet extends deep edge detection to category-aware semantic edges with an end-to-end network producing separate edge maps for defined semantic categories.The network treats category associations as multi-label outputs rather than restricting each pixel to one category.
- 1. Introduction: Edge detection supports tasks including segmentation, object proposals, 3D shape recovery, and 3D reconstruction.The paper motivates improved semantic edges as useful cues or constraints for broader scene-understanding problems.
- 1.2. Contributions: The proposed multi-label framework allows improved edge learning than a traditional multi-class framework.This directly addresses edge pixels that can belong to multiple semantic categories simultaneously.
- 1.2. Contributions: CASENet uses a nested ResNet architecture without deep supervision on bottom features, using those features to augment top classifications.The paper contrasts this design with architectures that supervise bottom convolutional sides directly.
- 1.2. Contributions: CASENet outperforms previous state-of-the-art methods by significant margins on the SBD and Cityscapes datasets.The evaluation includes SBD and extends semantic edge detection benchmarking to Cityscapes.
2. Problem Formulation
The paper formulates category-aware semantic edge detection as prediction of category-specific edge maps. Because edge pixels can belong to multiple categories, it uses multi-label targets and a class-imbalance-aware loss.
- 2. Problem Formulation: Given an input image I and K semantic categories, the task is to compute K edge maps with the same spatial size as I.Each map corresponds to one predefined semantic category.
- 2. Problem Formulation: The network output Y_k(p|I, W) represents the computed edge probability for category k at pixel p.The output is bounded between 0 and 1 and depends on the image and network parameters W.
- 2.1. Multi-label loss function: The formulation allows one edge pixel to belong to multiple semantic categories simultaneously.This follows from contours separating multiple classes and junctions involving three or more categories.
- 2.1. Multi-label loss function: The paper therefore treats category-aware semantic edge detection as multi-label rather than multi-class learning.The distinction reflects the possibility of simultaneous category membership at a single pixel.
- 2.1. Multi-label loss function: Each ground-truth label image Ȳ_k is a binary image indicating semantic edges for category k.The collection of K binary label images supplies supervision for the category-specific edge maps.
- 2.1. Multi-label loss function: The proposed multi-label loss accounts for the imbalance between edge and non-edge pixels using β.β denotes the percentage of non-edge pixels in the image and compensates for skewed sample numbers.
3. Network Architecture
CASENet is an end-to-end semantic edge detector that combines high-level classification with lower-level edge features through a nested skip-layer architecture. It addresses limitations of straightforward and deeply supervised designs by placing classification at the top and sharing bottom features across category activations.
- CASENet architecture: CASENet uses a modified ResNet-101 fully convolutional backbone that preserves low-level edge information while maintaining receptive field sizes.The first and fifth convolution blocks change stride from 2 to 1, with dilation added to subsequent layers.
- Basic architecture: The Basic architecture produces K non-mutually-exclusive semantic edge maps by applying a 1×1 classification layer, grouped up-sampling, and sigmoid probabilities.Each output map corresponds to one predefined semantic category and has the same spatial size as the input image.
- Deeply supervised architecture: The deeply supervised design adds K-channel classification modules to five residual-block outputs, fuses their activations, and applies six losses for supervision.Its sliced concatenation forms a 5K-channel activation map before grouped convolution produces the final K-channel output.
- Deeply supervised architecture: Sliced concatenation with grouped convolution integrates corresponding class-specific activations from different scales into the final fused activations.The authors state that experiments empirically support this design choice.
- CASENet architecture: CASENet replaces bottom-side classification with feature extraction, supervises classification only at the top, and uses shared rather than sliced concatenation.Bottom features are intended to augment top classifications while avoiding semantic classification at stages with limited receptive fields.
- CASENet architecture: Shared concatenation replicates the same bottom feature set for each of the K top activations, then grouped 1×1 convolution produces a K-channel fused activation map.This design lets lower-level features participate in higher-level semantic classification while retaining detailed edge localization and structure information.
4. Experiments
Experiments evaluate CASENet and baselines on SBD and Cityscapes using benchmark metrics, ablations, visualization, and architecture variants. CASENet outperforms prior methods and baselines, while results support multi-label learning and Side-5 supervision.
- Results on SBD: CASENet outperforms previous category-wise semantic edge detection methods on SBD using benchmark MF scores.The evaluation reports MF because AP is not well defined under the benchmark’s postprocessing behavior, and raw edge maps are used without postprocessing.
- Ablation studies: Using softmax instead of the proposed multi-label loss causes significant performance degradation on both VGG and ResNet.This comparison supports formulating category-aware semantic edge detection as multi-label rather than multi-class learning.
- Ablation studies: CASENet frequently achieves higher per-class MF and final mean MF than Basic and DSN architectures, indicating that bottom-side deep supervision may be unnecessary.The authors associate reduced distinctions across different supervision schemes partly with annotation quality and report that the distinction is more obvious on Cityscapes.
- Ablation studies: Adding Side-5 supervision improves performance and produces sharper semantic activations than CASENet−.The authors suggest Side-5 supervision helps higher-level features focus on semantic classification with less influence from bottom-layer interactions.
- Architecture variants: CASENet-VGG performs comparably to HFL-FC8 and significantly outperforms the reported HED baselines, while HFL-CRF performs slightly better with CRF postprocessing.The comparison indicates that the learning framework remains effective when instantiated with VGG, despite HFL using separate VGG networks for localization and classification.
- Architecture variants: Keeping stride 2 in res1 downgrades performance for lower input resolution, while binary edge losses on Side-1–3 appear divergent from Side-5 semantic classification.These experiments examine resolution and supervision choices beyond the main CASENet design.
- Results on Cityscapes: Cityscapes evaluation reports MF and AP, with CASENet consistently outperforming DSN across all classes by a significant margin.Cityscapes provides higher-quality annotations and more challenging, overlapping-object scenes that produce more multi-label boundary pixels.
5. Concluding Remarks
The paper presents CASENet as an end-to-end network for category-aware semantic edge detection and reports improvements over existing architectures and state-of-the-art methods.
- CASENet is an end-to-end deep network for category-aware semantic edge detection.
- CASENet improves over existing architectures used in edge detection and segmentation.
- The proposed multi-label learning framework leads to better learning behavior on edge detection.
- CASENet improves over previous state-of-the-art methods with significant margins.
A. Multi-label Edge Visualization
The paper visualizes multi-label semantic-edge predictions by encoding class confidences as HSV colors, with hue mixtures representing class combinations and prediction errors.
- Each semantic class receives a unique hue value for visualizing category-aware edge predictions.
- Pixel saturation is set from the maximum predicted class confidence across the K output channels.
- Testing edge responses are thresholded at 0.5, and the two strongest classes determine the hue.
- Hue mixing encodes multi-label edge types for the 19 Cityscapes semantic classes.
- Incorrect response strengths or false positives shift hue values away from ground truth, making prediction errors visible.
B.1. Early stage loss analysis
Experiments examine early losses, training-iteration stability, validation-based model selection, and qualitative class-wise predictions on SBD.
- CASENet’s fused loss initially exceeds its Side-5 loss, then drops faster and becomes consistently lower.
- CASENet produces sharper edges and stronger responses on difficult SBD edges than Basic and DSN.
- Small objects generate frequent false positives, including a sheep prediction on a rock.
- Raw-network precision-recall evaluation shows CASENet slightly but consistently outperforming the baselines.
- CASENet consistently outperforms Basic and DSN across evaluations from 16000 to 30000 iterations.
- 71.79% MF is obtained by CASENet versus 71.22% for Basic on the held-out SBD test set.
C.1. Additional qualitative results
Additional qualitative results compare class-wise semantic-edge predictions on SBD and Cityscapes, with CASENet consistently outperforming DSN on Cityscapes.
- Additional qualitative results are available through released videos on the Cityscapes validation set.
- Cityscapes comparisons show CASENet better detects challenging objects and DSN produces slightly more false positives on non-edge pixels.
- SBD visualizations cover class-wise predictions for 20 semantic categories across multiple methods.
- SBD qualitative results include difficult or failure cases for aeroplane, dining table, and sheep.
- Precision-recall curves are provided for each semantic class on SBD and for CASENet versus DSN on Cityscapes.