Source-linked AI summary
Active Boundary Loss for Semantic Segmentation
Chi Wang, Yunke Zhang, Miaomiao Cui, Peiran Ren, Yin Yang, Xuansong Xie, XianSheng Hua, Hujun Bao, Weiwei Xu
TL;DR
Semantic segmentation can retain errors at object boundaries, particularly for small and thin objects. This paper introduces ABL, which uses differentiable direction prediction to align predicted boundaries with ground truth during training. The authors report improved boundary details and segmentation metrics across image and video settings.
Problem
Segmentation errors remain at object boundaries, especially for small and thin objects, and commonly used cross-entropy loss does not explicitly enforce boundary alignment.
Source: Introduction · pp. 1–2Method
ABL detects predicted boundaries from current segmentation results and uses differentiable direction-vector prediction to move them toward ground-truth boundaries during end-to-end training.
Source: Introduction · p. 1, Active Boundary Loss · p. 3Results
Training with ABL improves boundary details and, across evaluated segmentation settings, improves boundary F-score and mean Intersection-over-Union.
Source: Introduction · p. 2, Conclusion · p. 7Takeaways & Limitations
ABL can be added to segmentation-network training to improve boundary details without changing the network structure at inference.
Source: Introduction · p. 2The authors identify reducing conflicts in the loss as future work and also plan to explore boundary-aware losses for depth prediction.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
This paper proposes a novel active boundary loss for semantic segmentation. It can progressively encourage the alignment between predicted boundaries and ground-truth boundaries during end-to-end training, which is not explicitly enforced in commonly used cross-entropy loss. Based on the predicted boundaries detected from the segmentation results using current network parameters, we formulate the boundary alignment problem as a differentiable direction vector prediction problem to guide the movement of predicted boundaries in each iteration. Our loss is model-agnostic and can be plugged in to the training of segmentation networks to improve the boundary details. Experimental results show that training with the active boundary loss can effectively improve the boundary F-score and mean Intersection-over-Union on challenging image and video object segmentation datasets.
Introduction
Semantic segmentation produces blurred boundaries despite advances in accuracy, and substantial errors remain, especially on small and thin objects. The paper proposes ABL, which progressively aligns predicted and ground-truth boundaries during end-to-end training and improves segmentation results alongside existing losses.
Introduction
- Boundary errors persist, especially for small and thin objects, despite boundary-aware approaches that have improved segmentation accuracy.The introduction calls for further study of the mutual dependence between semantic segmentation and boundary detection.Source: Introduction · p. 1
- ABL progressively aligns predicted boundaries with ground-truth boundaries during end-to-end training.Predicted boundaries are detected from the segmentation results produced by the current network.Source: Introduction · pp. 1–2
- ABL formulates boundary alignment as differentiable direction-vector prediction, moving predicted boundaries toward ground-truth boundaries in a probabilistic manner.The method also detaches gradient flow to suppress possible conflicts.Source: Introduction · p. 2
- Unlike cross-entropy and IoU losses, ABL supervises relationships between predicted- and ground-truth-boundary pixels, providing complementary information during training.The paper combines it with cross-entropy and Lovász-softmax loss; it is model-agnostic and can be added to image-segmentation training.Source: Introduction · p. 2
- Training with ABL improves boundary F-score and mean Intersection-over-Union on challenging image and video object-segmentation datasets.The evaluated networks include DeepLabV3, OCR, SwinTransformer, and STM; inference remains unchanged.Source: Introduction · p. 2
Related Work
Prior work improves segmentation through multi-scale feature fusion, boundary-aware training, and post-refinement, but boundary errors remain a concern for thin objects. ABL instead propagates ground-truth boundary information to regulate predicted boundaries without changing network structure.
Related Work
- Encoder-decoder FCNs and later methods fuse multi-scale features to improve localization accuracy and handle small objects.FCN-based segmentation methods also appear in video object segmentation, where temporal coherence and discriminative target features help address occlusion, appearance change, and fast motion.Source: Related Work · p. 2
- Boundary-aware methods add boundary-detection branches or control feature flow through pixel affinities near semantic boundaries.A frozen inverse transformation network has also been proposed for boundary-distance measurement under a homography assumption.Source: Related Work · p. 2
- DenseCRF and Segfix refine boundaries after segmentation, but may fail to model pixels inside thin objects with few interior pixels.The paper contrasts these methods with ABL, which encourages predicted- and ground-truth-boundary alignment and handles such boundaries well in its experiments.Source: Related Work · p. 2
- ABL propagates ground-truth-boundary information through a distance transform to regulate predicted boundaries while leaving network structure unchanged.The paper presents this as saving network-design effort.Source: Related Work · p. 2
- Unlike ABL, Boundary Loss is designed for unbalanced binary segmentation and is described as a regional IoU loss.The implementation couples ABL with an IoU loss to further refine boundary details.Source: Related Work · p. 2
Active Boundary Loss
ABL detects predicted boundaries from the current segmentation and uses ground-truth distances to choose directions that move them toward ground-truth boundaries. Its training loss converts those directions into a probabilistic target and suppresses conflicting gradients.
Active Boundary Loss
- ABL has two phases: it selects a candidate boundary neighbor closer to a ground-truth boundary, then uses KL divergence to guide predicted-boundary movement.The predicted boundaries are monitored as segmentation results change during training.Source: Active Boundary Loss · pp. 2–3
- ABL uses neighbor KL divergences as logits, increasing divergence toward the selected candidate while reducing it toward other neighbors.This probabilistic direction prediction is intended to push predicted boundaries toward ground-truth boundaries.Source: Active Boundary Loss · p. 3
- Predicted-boundary locations are detected from the current network's class-probability map using KL divergence between neighboring pixels.The method uses an adaptive threshold to keep detected boundary pixels below 1% of image pixels, which helps avoid misleading detections and reduce computation.Source: Active Boundary Loss · p. 3
- For each predicted-boundary pixel, ABL selects the neighboring pixel with the smallest ground-truth distance and encodes its direction as a one-hot target.The method dilates the predicted-boundary map by one pixel and applies this operation in the dilated region to accelerate movement.Source: Active Boundary Loss · p. 3
- The loss is weighted cross-entropy over predicted-boundary pixels, with ground-truth distance weighting deviations and pixels already on ground-truth boundaries discarded.The distance-weighting parameter θ is set to 20.Source: Active Boundary Loss · p. 3
- Neighboring predicted-boundary pixels can require opposing KL updates, producing contradictory gradients.ABL detaches gradients for neighboring pixels so only predicted-boundary pixels are adjusted; without detaching, the paper reports an approximately 3% mIoU drop.Source: Active Boundary Loss · p. 4
- Label smoothing sets the largest one-hot target probability to 0.8 and distributes 0.2/7 across the remaining directions.The paper says this regularizes ABL and can avoid over-confident updates when neighboring pixels share a distance value.Source: Active Boundary Loss · p. 4
Training Loss
The training objective combines cross-entropy with Lovász-softmax IoU loss and active boundary loss (ABL). Lovász-softmax helps retain small objects and balance noisy predicted boundaries, supporting ABL’s boundary refinement.
Training Loss
- Lovász-softmax loss is combined with cross-entropy and ABL to improve boundary details.The total training loss uses a weight for ABL.Source: Training Loss · p. 4
- Lovász-softmax loss uses the Lovász extension of the Jaccard loss for each class, based on class-wise prediction errors.The passage defines C as the number of classes and m(c) as the prediction-error vector for class c.Source: Training Loss · p. 4
- Lovász-softmax tends to prevent small objects from being ignored, preserving predicted boundaries that ABL needs to refine.Source: Training Loss · p. 4
- Lovász-softmax can balance noisy predicted boundary pixels, especially early in training.Source: Training Loss · p. 4
Experiments
The experiments evaluate ABL through loss ablations, image- and video-segmentation benchmarks, and qualitative boundary comparisons. They report improvements on several validation benchmarks and progressive boundary refinement.
Experiments
- The experiments use Cityscapes and ADE20K for image segmentation and evaluate ABL with DeepLabV3, OCR, and UperNet; STM is used for video object segmentation.Evaluation includes pixel accuracy, mIoU, and boundary F-score; the FKL comparison applies a KL-based loss at every image edge, not only semantic boundaries.Source: Experiments · pp. 4–5
Ablation Studies
- On Cityscapes, adding ABL to CE+IoU increased mIoU by 0.3%, while CE+IABL contributed a 1% improvement; on ADE20K, CE+IABL added around 0.65% over CE+IoU.Adding ABL only during the last 20% of training yielded a 0.1% mIoU improvement; the authors report that removing gradient detachment caused pixel-accuracy and mIoU drops.Source: Ablation Studies · pp. 5–6, Table 1 · p. 5, Table 2 · p. 5
Quantitative Evaluation
- ADE20K OCR+IABL gained 1.22% mIoU and 0.23% pixel accuracy; Cityscapes OCR gained 0.7% mIoU; DAVIS STM gained around 0.7% J-mean and 1% F-mean.The reported gains are over cross-entropy-only training for ADE20K, and on the DAVIS-2016 validation set for STM fine-tuning.Source: Table 3 · p. 5, Table 4 · p. 5, Quantitative Evaluation · pp. 6–7
- Against Segfix, IABL was comparable with DeepLabV3 and improved OCR mIoU by 0.3%; it scored higher at 3- and 5-pixel boundary dilations, but not 1 pixel.ABL+GDL also achieved higher DSC and smaller HD than BL+GDL on WMH, while IoU+ABL achieved higher Cityscapes mIoU than IoU+BL.Source: Quantitative Evaluation · pp. 6–7, Table 5 · p. 7, Table 6 · p. 7
Qualitative Results
- Qualitative examples show predicted boundaries moving toward ground-truth boundaries during DeepLabV3 training and improved boundary details after STM fine-tuning.The examples include Cityscapes and ADE20K image results and DAVIS-2016 video results.Source: Figure 4 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
Conclusion
The conclusion presents ABL as an end-to-end training loss that uses distance-transform boundary information to regulate predicted boundaries and improve semantic-segmentation boundary details. It identifies reducing loss conflicts and exploring depth prediction as future directions.
Conclusion
- Integrating ABL into network training substantially improves boundary details in semantic segmentation.Source: Conclusion · p. 7
- ABL uses a distance transform to propagate ground-truth boundary information and regulate network behavior at predicted boundaries.Source: Conclusion · p. 7
- Future work will investigate reducing loss conflicts around boundaries and designing boundary-aware losses for depth prediction.Source: Conclusion · p. 7