Source-linked AI summary
DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection
Gedas Bertasius, Jianbo Shi, Lorenzo Torresani
TL;DR
Contour detection has traditionally used low-level cues before higher-level recognition, although object recognition and contour prediction are closely related. DeepEdge inverts that pipeline with a multi-scale network using object-classification features and two independently trained prediction branches, achieving state-of-the-art contour detection results.
Problem
Traditional contour detection commonly uses low-level features before higher-level object detection, despite the close relationship between the two tasks.
Method
DeepEdge reuses the first five convolutional layers of KNet across four scales and attaches a bifurcated subnetwork with classification and human-agreement regression branches.
Results
DeepEdge achieves higher average precision than prior or concurrent work and state-of-the-art F-score in contour detection.
Takeaways & Limitations
Higher-level object features, multi-scale processing, and independently optimized classification and regression objectives improve contour prediction accuracy.
Takeaways & Limitations
The implementation was not optimized because the study focused primarily on the effective advantage of object-level features.
Abstract
from arXiv · showhide
Contour detection has been a fundamental component in many image segmentation and object detection systems. Most previous work utilizes low-level features such as texture or saliency to detect contours and then use them as cues for a higher-level task such as object detection. However, we claim that recognizing objects and predicting contours are two mutually related tasks. Contrary to traditional approaches, we show that we can invert the commonly established pipeline: instead of detecting contours with low-level cues for a higher-level recognition task, we exploit object-related features as high-level cues for contour detection. We achieve this goal by means of a multi-scale deep network that consists of five convolutional layers and a bifurcated fully-connected sub-network. The section from the input layer to the fifth convolutional layer is fixed and directly lifted from a pre-trained network optimized over a large-scale object classification task. This section of the network is applied to four different scales of the image input. These four parallel and identical streams are then attached to a bifurcated sub-network consisting of two independently-trained branches. One branch learns to predict the contour likelihood (with a classification objective) whereas the other branch is trained to learn the fraction of human labelers agreeing about the contour presence at a given point (with a regression criterion). We show that without any feature engineering our multi-scale deep learning approach achieves state-of-the-art results in contour detection.
1. Introduction
DeepEdge reverses the usual contour-to-object pipeline by using object-level features for contour detection. Its multi-scale architecture combines independently trained classification and human-agreement regression branches to achieve state-of-the-art results.
- Motivation: Contour detection and object recognition are treated as mutually related tasks, motivating a top-down contour-prediction pipeline.The method uses object-specific information as high-level cues rather than relying only on low-level contour cues.
- Method: DeepEdge learns contours directly from raw pixels while reusing the first five convolutional layers of the object-classification-trained KNet.These reused features incorporate object-level information into contour prediction.
- Method: The architecture processes four input scales simultaneously and connects their KNet features to a bifurcated learned subnetwork.This design combines local and global image information around candidate contour points.
- Learning objective: One branch uses edge classification, while the other regresses the fraction of human labelers agreeing about contour presence.The regression target is intended to better mimic the evidence used by human observers.
- Results: Fusing the two branch outputs yields state-of-the-art F-score and average precision in contour detection.The classification branch provides high recall and the regression branch provides high precision.
2. Related Work
The paper situates DeepEdge among low-level, global, and emerging deep-learning contour detectors. It contributes a multi-scale bifurcated CNN that learns directly from raw data and uses high-level object features to achieve state-of-the-art contour detection.
- Deep learning: DeepEdge demonstrates that convolutional networks can also perform lower-level contour detection, beyond their established use in high-level vision tasks.The paper reports state-of-the-art results on a contour detection task.
- Existing contour detectors: Earlier contour detectors include local methods based on small patches and global methods based on full-image information.Examples include sketch tokens and structured edges for local detection, and MCG, gPb, and sparse code gradients for global detection.
- Deep-learning approaches: Recent deep-learning approaches apply CNNs to contour prediction, including N 4 fields and DeepNet.N 4 fields combines dictionary learning and nearest neighbors within a CNN framework, while DeepNet uses a traditional CNN architecture.
- DeepEdge contributions: Its multi-scale bifurcated CNN avoids manual feature engineering by learning contours directly from raw data.The architecture is presented as a central contribution enabling state-of-the-art contour detection results.
- DeepEdge contributions: DeepEdge is presented as the first approach to use high-level object features for contour detection, thereby inverting the traditional low-level-cue pipeline.The authors report that this top-down approach yields state-of-the-art results.
3. The DeepEdge Network
DeepEdge combines fixed object-classification features, multi-scale processing, and two complementary prediction branches for contour detection. Candidate-centered patches are pooled across convolutional layers, while classification and regression outputs are fused into the final contour score.
- Single-Scale Architecture: Candidate contour points are selected with Canny, and fixed-size patches centered on them are used as network inputs.Boundary patches outside image limits are padded using mirror reflections.
- Single-Scale Architecture: The KNet reuses the first five convolutional layers of an ImageNet-trained object-classification network to extract location-preserving object-level features.The fully connected layers are excluded because they spatially scramble location information.
- Single-Scale Architecture: At each convolutional layer, centered feature-map sub-volumes are summarized with max, average, and center pooling to form hierarchical descriptors.Center pooling selects the activation corresponding to the candidate point, while the sub-volume sizes shrink in deeper layers.
- Single-Scale Architecture: The pooled features feed two separately trained fully connected branches: classification predicts contour presence, while regression predicts annotator agreement.The classification branch is less selective and favors recall; the regression branch models structural differences associated with agreement fractions and is more selective.
- Single-Scale Architecture: The two branch outputs are averaged at test time to produce the final contour score.The architecture uses the same fusion principle in the depicted single-scale network and in the trained bifurcated sub-network.
- Multi-Scale Architecture: Multi-scale processing runs four differently sized patches in parallel through the convolutional feature extractor, combining local and global image information.The patch sizes are 64 × 64, 128 × 128, 196 × 196, and full-image, each resized to 227 × 227 before KNet processing.
4. Experiments
Experiments on BSDS500 show that DeepEdge achieves state-of-the-art contour detection, with multi-scale, higher-level, pooling, and bifurcated objectives each contributing to performance. Qualitative results further show that the method suppresses weak internal contours while retaining object boundaries, although its implementation remains computationally costly.
- Comparison with Other Methods: DeepEdge achieves state-of-the-art BSDS500 contour detection according to F-score and average precision.
- Single Scale versus Multiple Scales: Multiple scales produce significantly higher F-scores and average precisions than single-scale settings.The architecture combines local and global information around candidate contour points.
- Advantages of Higher-Level Features: The fourth KNet convolutional layer provides the strongest individual contour predictions, while combining all layers yields state-of-the-art results.The fourth layer encodes higher-level information related to shape and object parts, while different layers provide complementary features.
- Pooling Schemes: Combining center, max, and average pooling improves contour detection beyond any single pooling technique.Center pooling performs best individually because the candidate contour point lies at the input patch center.
- Bifurcation and Training Objective: The classification branch provides high F-score, the regression branch high average precision, and their fusion improves both metrics.The branches are independently optimized for edge classification and annotator-agreement regression.
- Qualitative Results: Qualitative predictions assign lower probabilities to zebra stripes than to object boundaries, removing weak internal contours after thresholding.Using a probability threshold of 0.5 produces contours similar to the ground-truth map.
- Computational Cost: The current implementation requires about 60K KNet evaluations, and the authors did not optimize its implementation.They estimate about five minutes on a GPU and suggest future acceleration methods.
5. Conclusions
DeepEdge reverses the conventional contour-detection pipeline by using higher-level object cues in a multi-scale, bifurcated network. Experiments report substantial accuracy gains, higher average precision than prior or concurrent work, and successful application of pure CNNs to contour detection.
- DeepEdge inverts the conventional bottom-up pipeline by using higher-level object features to predict contours.
- Multiple image scales combine local and global information around candidate contour points, producing significantly better contour detection results.
- Independent optimization of contour classification and regression objectives further improves contour prediction accuracy.
- DeepEdge achieves higher average precision than any prior or concurrent work in the experiments.
- Pure CNN systems can be applied successfully to contour detection and possibly to other low-level vision tasks.