Source-linked AI summary
Devil is in the Edges: Learning Semantic Boundaries from Noisy Annotations
David Acuna, Amlan Kar, Sanja Fidler
TL;DR
Semantic boundary datasets contain noisy annotations because precise boundaries are laborious to produce, motivating sharper boundary prediction. STEAL combines a thinning layer and loss with level-set active alignment to learn from misaligned labels. It improves CASENet and other state-of-the-art methods while also refining coarse segmentation labels for more efficient annotation.
Problem
Semantic boundary prediction must identify object-class boundary pixels despite noisy labels produced by the high cost of precise annotation.
Method
STEAL combines a boundary thinning layer and loss with level-set active alignment, and can be added to existing learning-based boundary detectors.
Results
More than 4% MF(ODS) and 18.61% AP improvements over CASENet outperform current state-of-the-art methods, while coarse labels improve by more than 20% and 30% IoU for 16px and 32px errors.
Takeaways & Limitations
STEAL supports efficient dataset labeling by refining coarsely annotated segmentation masks and enabling annotators to draw few-click polygons.
Abstract
from arXiv · showhide
We tackle the problem of semantic boundary prediction, which aims to identify pixels that belong to object(class) boundaries. We notice that relevant datasets consist of a significant level of label noise, reflecting the fact that precise annotations are laborious to get and thus annotators trade-off quality with efficiency. We aim to learn sharp and precise semantic boundaries by explicitly reasoning about annotation noise during training. We propose a simple new layer and loss that can be used with existing learning-based boundary detectors. Our layer/loss enforces the detector to predict a maximum response along the normal direction at an edge, while also regularizing its direction. We further reason about true object boundaries during training using a level set formulation, which allows the network to learn from misaligned labels in an end-to-end fashion. Experiments show that we improve over the CASENet backbone network by more than 4% in terms of MF(ODS) and 18.61% in terms of AP, outperforming all current state-of-the-art methods including those that deal with alignment. Furthermore, we show that our learned network can be used to significantly improve coarse segmentation labels, lending itself as an efficient way to label new data.
1. Introduction
Semantic boundary detection targets object-class boundaries rather than generic image edges, but precise annotation is slow and noisy. STEAL addresses this by learning thin boundaries while modeling annotation misalignment, improving benchmark performance and refining coarse labels.
- Semantic boundary detection identifies pixels belonging to object-class boundaries rather than low-level image edges, supporting tasks such as instance segmentation, robot manipulation, and image editing.
- Precise boundary annotation can take 30–60s per object, causing datasets to contain noisy labels and detectors to produce undesirable thick boundaries.
- STEAL adds a layer and loss that enforce a maximum response along an edge normal, regularize direction, and directly optimize thin NMS edges.
- More than 4% MF(ODS) and 18.61% AP improvements over CASENet outperform current state-of-the-art methods.
- STEAL improves noisy segmentation labels by more than 20% IoU for 16px errors and 30% IoU for 32px errors, supporting efficient coarse annotation.
2. Related Work
Prior semantic boundary methods commonly thin predictions through postprocessing, while alignment-based methods address noisy labels with computational and structural limitations. STEAL instead optimizes thin edges during training and uses level sets to preserve coherent boundary alignment and refine coarse labels.
- Semantic Boundary Detection: STEAL directly optimizes non-maximum-suppressed boundaries during training instead of treating thinning solely as postprocessing.
- Semantic Boundary Detection: The method is an add-on to existing boundary architectures rather than a replacement edge-detection approach.
- Semantic Boundary Detection: Unlike SEAL’s computationally expensive assignment, STEAL’s level-set alignment preserves connectivity and proximity when inferring true object boundaries.
- Semantic Boundary Detection: STEAL handles substantial topology and deviation differences between annotated and inferred ground truth, enabling refinement of coarse segmentation labels.
- Level Set Segmentation: Level-set methods are used in segmentation because they can automatically handle topological changes such as splitting and merging.
3. The STEAL Approach
STEAL combines a backbone-agnostic boundary-thinning layer and loss with active alignment, jointly learning thin semantic edges and correcting noisy boundary labels through a level-set formulation.
- STEAL adds a backbone-agnostic boundary-thinning layer, augmented loss, and active-alignment framework for learning precise semantic edges from noisy annotations.The method jointly learns object boundaries while aligning noisy human annotations with latent true boundaries.
- 3.3. Thinning Layer and NMS Loss: The thinning layer samples predictions along each boundary normal and uses softmax normalization so the true boundary receives the maximum response.Five locations are sampled along each normal in the illustrated architecture, while the NMS loss targets a concentrated response at the boundary pixel.
- 3.3. Thinning Layer and NMS Loss: The direction loss regularizes predicted boundary normals toward ground-truth normals, complementing the NMS loss to produce crisp edges.The predicted normal is compared with the ground-truth normal in the angular domain.
- 3.4. Active Alignment: The level-set formulation preserves well-behaved boundaries, handles topological changes, and can evolve coarse masks toward predicted edges for improved segmentation labels.For coarse-to-fine annotation, the formulation sets λ = 0 and c = 1 while evolving the coarse mask.
- 3.4. Active Alignment: Active alignment jointly optimizes network parameters and latent boundary curves, using an energy that keeps inferred boundaries near annotations and high-probability predictions.The latent curves are initialized from noisy ground truth and evolved toward network-confident regions through iterative optimization.
4. Experimental Results
Experiments on SBD and Cityscapes evaluate STEAL under difficult boundary-matching protocols, compare it with existing boundary and segmentation methods, and test refinement of coarse labels. Across these settings, STEAL produces sharper boundaries, improves benchmark performance, and supports more accurate segmentation annotations.
- STEAL vs DeepLab-v3: At the strictest matching regime, STEAL’s boundaries perform 4.2% better than boundaries extracted from DeepLab V3+ segmentation masks.STEAL outperforms DeepLab edges across the reported matching thresholds despite using a lighter architecture.
- Results and Comparisons: Adding the NMS layer to CASENet outperforms SEAL by more than 1% in both MF(ODS) and AP, with active alignment improving performance further.SEAL is described as a method that explicitly addresses misaligned labels.
- Results and Comparisons: 5.35% MF(ODS) and 18.61% AP improvements over CASENet were obtained on the high-quality SBD test set, while the original test set gains were 3.72% and 17.11%.The evaluation uses both noisy and re-annotated SBD test sets.
- NMS Loss w/o Edge-NMS: Without test-time edge-NMS post-processing, the NMS loss still improves CASENet by 1.94% MF and 10.68% AP on the original dataset, and by 1.47% MF and 10.47% AP on the re-annotated dataset.This supports sharper raw predictions rather than gains arising only from inference-time post-processing.
- Qualitative Results: Qualitative SBD and Cityscapes results show crisper predictions than the base network, while active alignment illustrates refined true boundaries during training.The experiments include qualitative comparisons and visualizations of aligned boundaries.
5. Conclusion
The paper presents STEAL as a lightweight add-on for existing boundary detectors that learns thin boundaries while handling noisy annotations. Its experiments support both improved semantic boundary detection and refinement of coarse labels for more efficient dataset annotation.
- 5. Conclusion: STEAL combines a Thinning Layer and loss with active alignment to handle noisy annotations while improving semantic boundary detection.The approach is designed to work with existing boundary detectors.
- 5. Conclusion: The method tolerates substantial noise during training and inference and can refine coarse polygons drawn with only a few clicks into finer annotations.The conclusion frames this as an efficient way to label future datasets.