Source-linked AI summary
Holistically-Nested Edge Detection
Saining Xie, Zhuowen Tu
TL;DR
Detecting edges and object boundaries in natural images is fundamental but challenging because precise localization requires resolving perceptual ambiguity. HED addresses this with holistic image-to-image prediction and nested multi-scale feature learning, achieving state-of-the-art performance at 0.4 seconds per image using a GPU.
Problem
Precisely detecting edges and object boundaries in natural images is important across computer vision applications but requires resolving perceptual ambiguity across levels.
Method
HED uses a single-stream deep network with multiple side outputs for holistic image-to-image prediction and nested multi-scale feature learning under deep supervision.
Results
HED demonstrates state-of-the-art performance on natural images at practical speed, processing images in 0.4 seconds using a GPU.
Takeaways & Limitations
The results support combining multi-scale and multi-level visual responses for practical image-to-image edge detection without explicitly enforcing contextual and high-level information.
Abstract
from arXiv · showhide
We develop a new edge detection algorithm that tackles two important issues in this long-standing vision problem: (1) holistic image training and prediction; and (2) multi-scale and multi-level feature learning. Our proposed method, holistically-nested edge detection (HED), performs image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets. HED automatically learns rich hierarchical representations (guided by deep supervision on side responses) that are important in order to approach the human ability resolve the challenging ambiguity in edge and object boundary detection. We significantly advance the state-of-the-art on the BSD500 dataset (ODS F-score of .782) and the NYU Depth dataset (ODS F-score of .746), and do so with an improved speed (0.4 second per image) that is orders of magnitude faster than some recent CNN-based edge detection algorithms.
1. Introduction
The paper introduces HED, an end-to-end edge detector designed to resolve ambiguity in natural-image edges and object boundaries through holistic image-to-image prediction and nested multi-scale feature learning. It addresses limitations in prior CNN-based methods by targeting both detection quality and prediction speed.
- Prior limitations: SE achieves an F-score of .746 at 2.5 frames per second on BSD500, while recent CNN-based methods take several seconds to a few hours per prediction.The introduction identifies remaining room for improvement in both F-score performance and speed.
- Contribution: HED is an end-to-end system that automatically learns rich hierarchical features for natural-image edge and object-boundary detection.The method is intended to approach human ability to resolve ambiguity in edge and boundary detection.
- Contribution: HED performs holistic image-to-image prediction, directly converting an input image into an edge-map image using fully convolutional neural networks.The system does not explicitly model structured output but trains and predicts edges across the whole image.
- Contribution: HED performs nested multi-scale feature learning by applying deep-layer supervision to guide early classification results.This design is inspired by deeply-supervised nets and emphasizes inherited, progressively learned representations.
2. Holistically-Nested Edge Detection
HED frames edge detection as holistic image-to-image prediction with multi-scale, multi-level learning, distinguishing its holistically-nested architecture from alternative deep-learning configurations. It trains side outputs jointly with a learned weighted-fusion layer and uses class-balanced cross-entropy to address the severe edge/non-edge imbalance.
- Multi-scale learning: HED organizes multi-scale deep learning into four configurations: multi-stream, skip-net, multi-input single-model, and independent-network training.These categories clarify how the proposed holistically-nested approach differs from prior efforts.
- Holistically-nested networks: The holistically-nested network produces predictions from multiple scales while reducing representational and computational redundancy relative to existing approaches.The architecture is presented as a relatively simple variant for multi-scale prediction.
- Training formulation: The method treats each image holistically and independently, learning features that produce edge maps approaching the ground truth.The formulation uses M side-output layers, each paired with a classifier.
- Class-balanced training: 90% of a typical natural image’s ground truth is non-edge, creating a heavily biased edge/non-edge pixel distribution.This imbalance motivates the class-balancing strategy used in training.
- Class-balanced training: The training objective uses a per-pixel class-balancing weight β to offset the imbalance between edge and non-edge labels.The paper defines β = |Y−|/|Y| and 1 − β = |Y+|/|Y|.
- Prediction fusion: A learned weighted-fusion layer directly combines side-output predictions through a jointly learned fusion weight.The fusion layer has its own loss, Lfuse.
3. Network Architecture
HED adopts a modified, pretrained VGGNet architecture to provide deep, multi-level, multi-scale representations for edge detection. Its deep supervision guides side outputs toward progressively coarse-to-fine, local-to-global predictions, unlike weighted-fusion-only training.
- Network architecture: HED uses VGGNet initialized from image-classification pretraining, modified to connect side outputs to conv1_2, conv2_2, conv3_3, conv4_3, and conv5_3.The architecture is selected to support deep hierarchical features and multiple stages with different strides while easing training of the deep network.
- FCN and skip-layer architecture: Unlike HED, FCN uses only one output loss and does not explicitly produce multi-scale output predictions, despite combining coarse and fine information through skip connections.The FCN skip-net topology is described as a directed acyclic graph combining high-layer coarse information with low-layer fine information.
- Deep supervision: Deep supervision makes side-output predictions progressively coarse-to-fine and local-to-global while preserving critical object boundaries.Examples show side outputs becoming coarser and more global across layers, with important boundaries retained.
- Deep supervision: Without deep supervision, weighted-fusion-only training produces side outputs lacking discernible order, omits many critical higher-layer edges, and severely degenerates average precision.The benchmark F-score changes only marginally under the same experimental setup, despite the degradation in average precision.
4. Experiments
HED is implemented by fine-tuning a pre-trained VGG-16 network within Caffe, while side-output supervision and late merging improve multi-scale edge prediction. It achieves strong results on BSDS500 and NYUD, with additional training data raising BSDS500 performance to ODS=.797 (±.003).
- Implementation: HED is built in Caffe on FCN and DSN implementations, with the full network fine-tuned from a pre-trained VGG-16 initialization.The implementation requires relatively little additional engineering.
- Side outputs: Combining side-output predictions from multiple scales improves performance, and every side-output layer contributes to the final result.All side-output predictions are obtained in one pass, enabling configuration comparisons without extra computation.
- Late merging: Late merging of the fusion output with side outputs yields the best performance across all reported measures without extra cost.Weighted fusion gives the best F-score, while averaging side outputs gives better average precision before merging.
- BSDS500: 0.797 (±.003) ODS is achieved on BSDS500 after adding only 100 training images, improving from ODS=.782 and nearly reaching the human benchmark.The original BSDS500 evaluation uses 200 training, 100 validation, and 200 testing images.
- NYUD: .746 ODS is achieved on NYUD by averaging predictions from RGB and HHA models, with Average2-4 used during testing.HHA encodes depth as horizontal disparity, height above ground, and local surface-normal angle relative to gravity.
5. Conclusion
The paper develops a convolutional-neural-network-based edge detection system that achieves state-of-the-art performance on natural images at practically relevant speed, building on fully convolutional and deeply supervised networks.
- Conclusion: The system achieves state-of-the-art performance on natural images at practical speed, processing images in 0.4 seconds on GPU and 12 seconds on CPU.These reported runtimes establish practical relevance across GPU and CPU settings.
- Conclusion: The method combines fully convolutional neural networks with deeply-supervised nets for edge detection.Its network structure and parameters are initialized using a pre-trained trimmed VGGNet.
A. More Results
The updated HED experiment triples the training data through multi-scale augmentation and uses full-resolution images, yielding stronger BSDS500 results while narrowing the F-score gap between models with and without deep supervision.
- Updated experiment setting: Training images are scaled to 50%, 100%, and 150%, tripling the augmented dataset, while training uses full-resolution images instead of 400 × 400 resizing.These changes define the newly retrained model’s experiment setting.
- Deep supervision: Under the new settings, the F-score gap narrows between models with and without deep supervision, while qualitative and quantitative observations remain similar to Section 3.2.The authors report consistency with the earlier analysis despite the narrower gap.
- BSDS500 results: The proposed HED framework achieves ODS=.790, OIS=.808, and AP=.811, while the late-merging variant achieves ODS=.788, OIS=.808, and AP=.840.HED has the best F-score, whereas late merging has the best average precision.