Source-linked AI summary
Pushing the Boundaries of Boundary Detection using Deep Learning
Iasonas Kokkinos
TL;DR
Boundary detection is difficult because human annotations can disagree about boundary locations, yet it remains important for computer vision and downstream tasks. The paper adapts deep DCNN training with improved losses, graduated supervision, multi-resolution training, external data, and grouping. The resulting detector exceeds reported human F-measure on BSD and improves semantic-segmentation performance, while the authors identify further gains from additional cues and high-level-task exploration.
Problem
Boundary detection is a difficult low-level vision task because human annotations can be inconsistent about boundary locations.
Method
The paper combines improved boundary-detection losses, graduated deep supervision, multi-resolution training with external data, and grouping integration within a deep architecture.
Results
0.813 BSD F-measure exceeds the reported human score of 0.803, and integrating boundary information into DenseCRF improves semantic segmentation from 74.8 to 75.4.
Takeaways & Limitations
The detector is fully integrated in Caffe, processes a 320x420 image in less than one second, and supports improvements in higher-level vision tasks.
Takeaways & Limitations
The paper reports that corrected object-proposal improvements are not large enough to be considered substantial and anticipates further exploration of additional cues and high-level tasks.
Abstract
from arXiv · showhide
In this work we show that adapting Deep Convolutional Neural Network training to the task of boundary detection can result in substantial improvements over the current state-of-the-art in boundary detection. Our contributions consist firstly in combining a careful design of the loss for boundary detection training, a multi-resolution architecture and training with external data to improve the detection accuracy of the current state of the art. When measured on the standard Berkeley Segmentation Dataset, we improve theoptimal dataset scale F-measure from 0.780 to 0.808 - while human performance is at 0.803. We further improve performance to 0.813 by combining deep learning with grouping, integrating the Normalized Cuts technique within a deep network. We also examine the potential of our boundary detector in conjunction with the task of semantic segmentation and demonstrate clear improvements over state-of-the-art systems. Our detector is fully integrated in the popular Caffe framework and processes a 320x420 image in less than a second.
1 INTRODUCTION
Boundary detection is a foundational but intrinsically ambiguous vision task because plausible segmentations depend on the intended use. This work applies deep learning to narrow the human–machine gap and combines boundary detection with grouping and higher-level vision tasks.
- Problem: Boundary detection is a cornerstone computer-vision problem whose labels can vary with the task and produce inconsistent human segmentations.The same image may plausibly be segmented differently for different purposes, such as recognizing a checkerboard versus carrying it as one object.
- Problem: Human boundary annotations are difficult to benchmark because annotators may disagree about which details count as boundaries.The BSD evaluates each annotator against a committee of the others using precision, recall, and F-measure.
- Results: 0.813 F-measure is achieved by the system on BSD, compared with 0.803 for human annotators.These values refer to the task’s F-measure under the reported evaluation.
- Results: 0.8308 F-measure is obtained under Optimal Image Scale, while the common-dataset-threshold Optimal Dataset Scale score is 0.813.The image-specific oracle sets a separate threshold for each image.
- Contributions: The paper contributes improved deep-learning training, grouping integration, and applications to semantic segmentation and region proposal generation.These advances are presented as three contribution fronts.
2 HED AND DSN TRAINING
HED adapts deep supervision to boundary detection by training intermediate DCNN layers through side outputs and combining them through a learned late-fusion stage. Its loss balances positive and negative boundary samples while multi-layer predictions provide multi-scale information.
- HED and DSN: HED fine-tunes a VGG DCNN for boundary detection using Deep Supervised Network training.DSN informs intermediate layers about the final classification objective.
- Architecture: Side parameters steer intermediate layers toward features useful for classification even when those layers operate independently.This gives intermediate representations preliminary task-specific predictive information.
- Architecture: Each intermediate DCNN layer feeds a side layer that is independently penalized, and the side outputs are combined in a jointly trained late-fusion stage.The architecture applies the loss to both side outputs and their final combination.
- Loss: HED uses a class-balanced cross-entropy loss to address the substantially larger number of negative image samples.The weighting parameter β mitigates the imbalance between positive and negative training indices.
- Fusion: The late-fusion score is a learned linear combination of side-layer predictions that calibrates their relative importance and exploits multi-scale information.The fusion weights combine the fields predicted by the individual side layers.
- Optimization: The overall HED objective sums side-layer and fusion losses and is optimized with stochastic gradient descent with momentum.The side terms encourage discriminative intermediate layers, while the fusion term calibrates their contributions.
3 IMPROVED DEEP BOUNDARY DETECTION TRAINING
The paper improves deep boundary detection by addressing annotation uncertainty, refining supervision and multi-resolution training, and incorporating grouping and external data. These changes are evaluated incrementally against a HED-type baseline using BSD performance measures.
- Three advances guide the training improvements: Multiple Instance Learning, Graduated Deep Supervision, and multi-scale training with external data.The work also incorporates classical grouping ideas and applies the detector to higher-level vision tasks.
- 3.1 DEALING WITH ANNOTATION INCONSISTENCIES: Annotation uncertainty makes pixel-level boundary labels difficult because semantically agreeing annotators may place boundaries at different locations.The method addresses this positional ambiguity rather than requiring every nearby pixel to share one fixed label.
- 3.1 DEALING WITH ANNOTATION INCONSISTENCIES: Multiple Instance Learning labels a bag positive when at least one nearby instance supports the boundary, while forcing negative samples to remain negative.Boundary bags are formed from positions near each ground-truth boundary, within a distance threshold and associated spatial constraints.
- 3.1 DEALING WITH ANNOTATION INCONSISTENCIES: The MIL variant with distance d = 1 improves performance over the baseline, while d = 2 yields similar improvements.These improvements are reported in the MIL column of Table 1.
- 3.2 GRADUATED DSN TRAINING: Graduated Deep Supervision decreases the side-loss weight over training while keeping the fusion-loss weight fixed, transitioning from DSN training toward a skip-layer architecture.This lets the final fusion layer use side outputs without retaining the compromises required to keep all side losses low.
- 3.3 MULTI-RESOLUTION ARCHITECTURE: Multi-resolution training processes differently scaled images with tied weights, fuses their outputs, and improves boundary detection while reducing over-fitting and accelerating convergence.The architecture uses a three-level pyramid and an additional fusion layer combining the individual-resolution results.
4 USING GROUPING IN A DEEP ARCHITECTURE
The detector incorporates grouping through Normalized Cuts, combining global spectral boundary evidence with multi-resolution network probabilities to improve boundary detection beyond human performance.
- Grouping with Normalized Cuts: The detector models pixels as a weighted graph whose affinities use the Intervening Contour cue estimated by the boundary network.Normalized Cuts then produces a generalized eigenvector problem whose embeddings encode inter-pixel similarity.
- Grouping with Normalized Cuts: GPU integration of a Lanczos solver addresses the roughly 60-second CPU cost of computing 10 eigenvectors for a 321 × 481 image.The implementation integrates the Damascene GPU solver with Caffe.
- Results: 0.813 F-measure results from linearly combining multi-resolution network posteriors with spectral boundary magnitude, surpassing the earlier 0.807 and human 0.803.The spectral probability-of-boundary cue supplies global evidence through directional derivatives of the embeddings.
- Results: The improvement sequence starts slightly below HED and produces a detector whose F-measure exceeds human performance.Table 2 compares the resulting system with state-of-the-art boundary detection while retaining speed above 1 frame per second.
5 SYNERGY WITH SEMANTIC SEGMENTATION
The boundary detector is applied to semantic segmentation by combining a multi-resolution architecture and DenseCRF inference with boundary-derived Normalized Cut features and graph-cut processing.
- Segmentation framework: The fully convolutional boundary model can be combined with FCNN-based semantic segmentation and DenseCRF methods.DenseCRF adds discriminative power to FCNN predictions through image-based pairwise terms.
- DenseCRF formulation: DenseCRF assigns pixel-label configurations an energy composed of CNN-based unary probabilities and bilateral filter-like pairwise potentials.The pixel-label assignment is represented by x, while E(x) denotes the energy function.
- Scope caveat: An earlier report’s large object-proposal improvement was caused by an erroneous baseline calculation; after correction, the remaining gains were not substantial.This correction limits the strength of the object-proposal result claimed in the earlier version.
- Multi-resolution segmentation: The segmentation model adapts tied-weight multi-resolution processing and late score fusion, improving over its single-resolution counterpart.The supplied passage reports improvements in Table 3 but truncates the complete comparison values.
- Boundary integration: Normalized Cut eigenvectors augment RGB features in DenseCRF, conveying boundary-based proximity into inference.The eigenvectors increase the feature dimensionality from 3 to 6 before graph-cut processing.
- Results: Performance increases from 74.8 to 75.4 after adding Normalized Cut eigenvectors, then to 75.7 with boundary-sensitive graph-cut inference.The reported results are on the VOC 2012 semantic segmentation setting summarized in Table 3.
6 CONCLUSION
The paper presents a faster deep-learning boundary detector that exceeds human F-measure on BSD and identifies additional low-level cues and high-level tasks for future investigation.
- Conclusion: The proposed method substantially improves deep learning-based boundary detection and operates in less than one second per frame within Caffe.Its BSD F-measure is higher than that of humans.
- Future work: The authors anticipate further gains from jointly treating symmetry, surface orientation, and depth, and plan to explore object detection and recognition.These are identified as directions for extending the detector beyond the reported work.