Source-linked AI summary
InverseForm: A Loss Function for Structured Boundary-Aware Segmentation
Shubhankar Borse, Ying Wang, Yizhe Zhang, Fatih Porikli
TL;DR
Semantic segmentation losses based on pixel labels do not adequately capture spatial transformations between predicted and target boundaries. The paper introduces InverseForm, an inverse-transformation distance loss integrated with existing segmentation losses and models. Across three benchmarks and single-task and multi-task settings, it consistently outperforms baselines and reports state-of-the-art results on NYU-Depth-v2 and PASCAL without added inference cost.
Problem
Cross-entropy-based losses ignore spatial distances between boundary pixels, making them inadequate for measuring localized translations, rotations, scaling, and other boundary transformations.
Method
InverseForm uses an inverse-transformation network to estimate parametric transformations between boundary maps and combines the resulting distance loss with pixel-based segmentation loss.
Results
The method consistently outperforms baselines across Cityscapes, NYU-Depth-v2, and PASCAL settings, including a 0.7% mIoU gain over the SA-Gates baseline on NYU-Depth-v2.
Takeaways & Limitations
InverseForm is a plug-in boundary-aware loss that improves segmentation and structured boundary outputs without additional inference computational load.
Takeaways & Limitations
The authors report remaining room for improvement and continue seeking an optimal architecture and distance measure.
Abstract
from arXiv · showhide
We present a novel boundary-aware loss term for semantic segmentation using an inverse-transformation network, which efficiently learns the degree of parametric transformations between estimated and target boundaries. This plug-in loss term complements the cross-entropy loss in capturing boundary transformations and allows consistent and significant performance improvement on segmentation backbone models without increasing their size and computational complexity. We analyze the quantitative and qualitative effects of our loss function on three indoor and outdoor segmentation benchmarks, including Cityscapes, NYU-Depth-v2, and PASCAL, integrating it into the training phase of several backbone networks in both single-task and multi-task settings. Our extensive experiments show that the proposed method consistently outperforms baselines, and even sets the new state-of-the-art on two datasets.
1. Introduction
The paper introduces InverseForm, a boundary-distance loss that complements pixel-based losses by modeling spatial transformations between predicted and target boundaries. It integrates into existing single-task and multi-task segmentation models without added inference cost and improves benchmark performance.
- Motivation: Cross-entropy overlooks boundary-pixel distances and cannot effectively measure translations, rotations, or scaling between predicted and target boundaries.The authors identify this limitation as a motivation for a spatially aware loss.
- Method: InverseForm uses an inverse transformation network to learn parametric transformations between local boundary regions and measure their spatial distance.The network models the distance between boundary maps and complements pixel-label accuracy during training.
- Method: The method plugs into existing segmentation backbones and adds no inference size, computational, or memory cost.It is described as architecture-agnostic, plug-and-play, and compatible with multi-task learning.
- Experiments: Experiments cover NYU-Depth-v2, Cityscapes, and PASCAL across single-task and multi-task settings.The study compares the method with contemporary segmentation and multi-task approaches on these benchmarks.
- Results: The method consistently outperforms baselines and state-of-the-art methods in reported single-task and multi-task evaluations.The paper reports state-of-the-art performance on NYU-Depth-v2 and PASCAL settings.
2. Related Work
Prior boundary-aware segmentation methods commonly use weighted cross-entropy, which does not adequately represent spatial boundary changes. InverseForm instead models spatial distance through an inverse-transformation approach that captures broader boundary transformations.
- Boundary-aware segmentation: Prior boundary-aware methods include global energy models, feature propagation, and multi-task formulations for enhancing segmentation with boundary information.The related work describes several approaches to modeling boundary-aware segmentation.
- Loss functions: Weighted cross-entropy focuses on pixel labels while ignoring boundary distance, limiting its ability to model shifts, scales, and rotations.The paper identifies this as a central difference between its method and previous work.
- InverseForm: InverseForm uses an inverse-transformation network to model spatial distance between boundary maps rather than estimating only discrete offset maps.The network regresses homography parameters and derives a distance measure from them.
- Multi-task learning: Multi-task learning shares representations across supervisory signals while exploring task interactions and saving memory and computation.The related work situates segmentation, depth, and other task combinations within this framework.
3. Proposed Scheme: InverseForm
InverseForm measures spatial transformations between boundary maps with an inverse-transformation network, addressing pixel-wise losses’ inability to represent boundary distance. It is integrated with existing segmentation losses through tiled boundary comparisons and can be used without added inference computation.
- Motivation: Pixel-wise cross-entropy ignores spatial distance between boundaries, making it inadequate for localized translations, rotations, and scaling.Small boundary displacements can receive disproportionately high penalties, encouraging thicker and distorted predicted boundaries.
- Motivation: Existing alternatives are limited: Hausdorff distance is inefficient for semantic segmentation, while correlation models translations but not other transformations.The method therefore targets an efficient distance function for general boundary transformations.
- Inverse transformation network: InverseForm inputs two boundary maps and predicts homography transformation parameters, functioning as the inverse operation of a spatial transformer network.The network uses dense layers to model spatial distance rather than serving as an unsupervised-learning encoder.
- Distance measures: After training, the inverse-transformation network is frozen and converts boundary-map agreement into a spatial distance, with identity representing a perfect match.The proposed distances include Euclidean and geodesic formulations over homography parameters.
- Distance measures: The Euclidean formulation models shifts and scale relations but fails to reflect rotations and other perspective transformations.The geodesic formulation instead treats homographies as transformations on an analytical manifold and uses an SO(3) projection for training.
- Loss integration: InverseForm splits predicted and ground-truth boundaries into tiles, measures each pair’s spatial distance, and integrates the resulting loss with pixel-based segmentation losses.The scheme is applied during training and can be added to existing backbones without increasing inference computation.
4. Experimental Results
Experiments on NYU-Depth-v2, PASCAL, and Cityscapes evaluate InverseForm across single-task and multi-task settings. The method consistently improves segmentation and boundary quality without extra inference computation, including reported gains over strong baselines.
- NYU-Depth-v2: InverseForm consistently improves NYU-Depth-v2 segmentation mIoU and boundary mBA across HRNet backbones in single-task and multi-task settings.Table 1 evaluates semantic segmentation, depth, edge detection, and surface normal estimation in multi-task settings.
- NYU-Depth-v2: 0.7% mIoU improvement over the SA-Gates baseline is reported on NYU-Depth-v2 using multi-scale RGBD inference.The model uses a ResNet-101 backbone and DeepLab-V3+ decoder.
- PASCAL: PASCAL experiments show consistent performance improvement after adding InverseForm to boundary detection in ASTMT and MTI-Net settings.The evaluation includes segmentation, saliency, human parts, surface normals, and boundary detection.
- Cityscapes: 0.3% and 1.1% mIoU gains over SegFix and Gated-SCNN are reported on Cityscapes, while adding InverseForm to GSCNN yields a 1.6% mIoU gain.All models trained with InverseForm consistently improve over their baselines, and the method requires no extra inference computation.
5. Ablation Studies
The ablation studies compare inverse-transformation architectures and distance formulations for InverseForm. Euclidean and geodesic distances show no clear winner, while geodesic distance can cause exploding gradients and restrict hyperparameter search.
- Inverse-transformation architecture: The study compares a dense inverse-transformation architecture with the convolutional architecture used in AET.Experiments use pretrained models fine-tuned on tiled NYU-Depth-v2 boundary images.
- Distance function: Experiments evaluate InverseForm with geodesic and Euclidean distance measures using vanilla HRNet-w48 and HRNet-w18 backbones.Both formulations are tested on NYU-Depth-v2 with the architecture defined in Figure 4.
- Distance function: Geodesic and Euclidean distance show no clear winner in the reported results.The authors continue exploring the optimal InverseForm network architecture and distance measure.
- Distance function: Geodesic distance can lead to exploding gradients, severely limiting the hyperparameter search space.Euclidean distance may model perspective homography less closely but provides a wider search space and more consistent improvement.
6. Conclusion
The paper concludes that InverseForm is a distance-based, boundary-aware segmentation method that improves backbone models without additional inference computation. It captures boundary transformations more effectively than cross-entropy-based measures, while the ablations leave room to optimize its architecture and distance measure.
- Conclusion: InverseForm consistently improves semantic segmentation backbones while adding no computational load during inference.Boundary detection and segmentation are jointly optimized during training, but only the backbone is used at inference.
- Conclusion: The distance-based measure captures boundary transformations more effectively than cross-entropy-based measures and produces better structured segmentation outputs.The conclusion reports superior segmentation accuracy alongside improved structure.
- Conclusion: Ablation experiments indicate room for improvement in the optimal InverseForm architecture and distance measure.The paper identifies these components as continuing research directions.