Source-linked AI summary
High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision
Gedas Bertasius, Jianbo Shi, Lorenzo Torresani
TL;DR
Boundary detection has traditionally relied on low-level cues, despite evidence that human judgments use object-level reasoning. The paper reuses object-classification features to detect High-for-Low boundaries and then applies them as Low-for-High cues, improving boundary detection and three high-level vision tasks. The resulting system is accurate, efficient, and applicable across these tasks.
Problem
Most boundary detectors rely on low-level features, although perception studies suggest that humans use object-level reasoning when judging boundaries.
Method
The method reuses object-level features from a pretrained classification network to detect High-for-Low boundaries and applies those boundaries to downstream vision tasks.
Results
The method achieves better results on boundary detection, semantic boundary labeling, semantic segmentation, and object proposal generation than the compared state-of-the-art methods.
Takeaways & Limitations
The paper presents an accurate and efficient boundary detector whose semantically meaningful boundaries improve three high-level vision tasks.
Takeaways & Limitations
BSDS500 evaluation includes orphan boundaries marked by only one or two annotators, although these comprise around 30% of the dataset and are often considered uninformative.
Abstract
from arXiv · showhide
Most of the current boundary detection systems rely exclusively on low-level features, such as color and texture. However, perception studies suggest that humans employ object-level reasoning when judging if a particular pixel is a boundary. Inspired by this observation, in this work we show how to predict boundaries by exploiting object-level features from a pretrained object-classification network. Our method can be viewed as a "High-for-Low" approach where high-level object features inform the low-level boundary detection process. Our model achieves state-of-the-art performance on an established boundary detection benchmark and it is efficient to run. Additionally, we show that due to the semantic nature of our boundaries we can use them to aid a number of high-level vision tasks. We demonstrate that using our boundaries we improve the performance of state-of-the-art methods on the problems of semantic boundary labeling, semantic segmentation and object proposal generation. We can view this process as a "Low-for-High" scheme, where low-level boundaries aid high-level vision tasks. Thus, our contributions include a boundary detection system that is accurate, efficient, generalizes well to multiple datasets, and is also shown to improve existing state-of-the-art high-level vision methods on three distinct tasks.
1. Introduction
The paper uses object-level features from a pretrained classification network to detect semantically meaningful boundaries, then applies those boundaries to several high-level vision tasks. It reports improved results across boundary detection and three downstream tasks while maintaining an efficient architecture.
- Most boundary detection methods use low-level color and gradient features despite evidence that humans use object-level reasoning when perceiving boundaries.
- The proposed High-for-Low approach uses object-level features as cues for low-level boundary detection.
- The architecture reuses features from sixteen convolutional layers of a VGG network pretrained for object classification.
- HFL boundaries improve three high-level vision tasks: semantic boundary labeling, semantic segmentation, and object proposal generation.
- Across boundary detection and the three high-level tasks, the method achieves better results than the compared state-of-the-art method for every presented evaluation metric.
- The system is presented as accurate, efficient, applicable to high-level vision tasks, and capable of detecting boundaries in near-real time.
2. Related Work
Prior boundary detectors largely rely on low-level cues or specialized learning architectures. This work instead uses object-level features and feature reuse to obtain strong boundary results with a simpler, near-real-time system.
- Contour detection methods are grouped into spectral, supervised discriminative, and deep-learning-based approaches.
- Spectral methods formulate contour detection as an eigenvalue problem, while discriminative methods use supervised classifiers such as SVMs or random forests.
- Deep-learning approaches include sparse-coding and CNN-based methods that reconstruct images, use dictionary learning, or directly predict contours.
- Unlike DeepEdge, the proposed method avoids a complicated multi-scale bifurcated architecture while achieving better results and near-real-time execution.
- The paper uses object-level information to learn semantic boundaries from human-annotated data without feature engineering.
3. Boundary Detection
The method detects boundaries efficiently by combining high-recall candidate contours with interpolated object-level features from VGG’s convolutional layers. Its HFL boundaries outperform prior methods on BSDS500, including consensus annotations, while emphasizing semantically meaningful object-background boundaries.
- Architecture: Candidate contour points reduce computation before the image is processed through VGG’s 16 convolutional layers.The system uses SE for efficient, high-recall candidate extraction and upsamples the image to reduce pooling-related information loss.
- Architecture: VGG convolutional features provide object-level information while preserving spatial detail needed for boundary localization.The model uses convolutional layers rather than fully connected layers, and feature visualizations show high activations around objects.
- Architecture: Feature interpolation produces a 5504-dimensional vector for each candidate point from a single image-wide convolutional pass.This avoids repeatedly processing overlapping patches and is identified as crucial for efficient prediction.
- Architecture: The predictor maps interpolated candidate-point features to human boundary agreement and accumulates them into a boundary probability map.Two fully connected layers are optimized against the fraction of annotators agreeing that a boundary is present.
- Evaluation: HFL boundaries outperform prior methods on both BSDS500 F-score measures for “any” ground-truth boundaries.The benchmark evaluates fixed contour threshold, per-image best threshold, and average precision measures.
- Evaluation: On the perceptually meaningful “consensus” set, HFL boundaries outperform or tie prior methods on all three metrics and run faster than earlier deep-learning systems.The consensus set contains boundaries marked by all human annotators, whereas orphan boundaries marked by only one or two annotators comprise around 30% of BSDS500.
- Analysis: Feature-importance visualization places many heavily weighted predictors in VGG’s deepest layers, which are most associated with object information.The linear regression is used only for visualization, not for the accuracy analysis.
- Evaluation: Qualitative comparisons show HFL detects object-background boundaries with high confidence when color variation is small.SE relies on low-level color and texture features and assigns low confidence in these cases, while HFL uses object-level features.
4. High-Level Vision Applications
HFL boundaries are used as low-level cues to improve semantic boundary labeling, semantic segmentation, and object proposal generation. Across these applications, the method improves or matches existing approaches while producing semantically meaningful boundaries.
- Semantic Boundary Labeling: The semantic boundary-labeling pipeline assigns object classes to predicted boundaries using concurrent boundary prediction and semantic segmentation outputs.It evaluates HFL-FC8 and HFL-CRF variants using pretrained FCN or DeepLab-CRF outputs.
- Semantic Boundary Labeling: HFL boundaries improve semantic boundary labeling across all 20 SBD object categories under both MF and AP.HFL-CRF performs better than HFL-FC8, consistent with the spatial coherence enforced by the CRF.
- Qualitative Results: Qualitative examples show that HFL identifies and localizes boundaries for multiple object classes appearing simultaneously.The visualization presents input images alongside semantic HFL boundaries for different object classes.
- Semantic Segmentation: HFL boundaries are incorporated into DeepLab-CRF through boundary-derived features intended to improve spatial coherence and segmentation accuracy.The features are generated from predicted HFL boundaries rather than object-labeled boundaries.
- Semantic Segmentation: The HFL features are applicable to generic CRF-based methods because they contribute directly to the CRF rather than to the underlying DeepLab network.This makes the feature augmentation compatible with improved DeepLab architectures.
- Object Proposal Generation: When used with EdgeBoxes, HFL boundaries produce better or equivalent object-proposal results than SE boundaries at IOU values of 0.65, 0.7, and 0.75.The comparison covers all three evaluation metrics at each reported IOU threshold.
5. Conclusions
The paper presents an efficient boundary detector that uses object-level features to produce semantically meaningful HFL boundaries. These boundaries also improve state-of-the-art methods across semantic boundary labeling, semantic segmentation, and object proposal generation.
- Contributions: Object-level features produce semantically meaningful boundaries that achieve higher F-score than prior boundary detection methods.The approach is framed as High-for-Low because high-level object features inform low-level boundary detection.
- Contributions: HFL boundaries improve state-of-the-art methods on semantic boundary labeling, semantic segmentation, and object proposal generation.This use of boundaries as cues for high-level tasks is framed as a Low-for-High scheme.
- Contributions: The boundary detection method is described as accurate, efficient, applicable to varied datasets, and useful for multiple high-level vision tasks.The authors also state that they plan to release the HFL source code upon publication.