Source-linked AI summary
Richer Convolutional Features for Edge Detection
Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Kai Wang, Xiang Bai
TL;DR
Edge detection needs representations rich enough to distinguish edges and boundaries across varied image content, while existing CNN architectures do not fully exploit their feature hierarchies. RCF combines convolutional features from all layers for holistic image-to-image prediction and reports strong BSDS500 accuracy at two operating speeds.
Problem
Edge detection requires representations that capture semantically meaningful boundaries and fine details, but existing CNN-based methods do not fully exploit rich convolutional feature hierarchies.
Method
RCF combines complementary convolutional features from all CNN layers in a holistic fully convolutional architecture for pixel-wise image-to-image edge prediction.
Results
0.811 ODS F-measure at 8 FPS is achieved on BSDS500, while a fast RCF version achieves 0.806 at 30 FPS.
Takeaways & Limitations
RCF is reported as accurate and efficient for edge detection and achieves competitive results when its edges are used for classical image segmentation.
Abstract
from arXiv · showhide
In this paper, we propose an accurate edge detector using richer convolutional features (RCF). Since objects in nature images have various scales and aspect ratios, the automatically learned rich hierarchical representations by CNNs are very critical and effective to detect edges and object boundaries. And the convolutional features gradually become coarser with receptive fields increasing. Based on these observations, our proposed network architecture makes full use of multiscale and multi-level information to perform the image-to-image edge prediction by combining all of the useful convolutional features into a holistic framework. It is the first attempt to adopt such rich convolutional features in computer vision tasks. Using VGG16 network, we achieve \sArt results on several available datasets. When evaluating on the well-known BSDS500 benchmark, we achieve ODS F-measure of \textbf{.811} while retaining a fast speed (\textbf{8} FPS). Besides, our fast version of RCF achieves ODS F-measure of \textbf{.806} with \textbf{30} FPS.
1 INTRODUCTION
Edge detection requires representations that capture both semantically meaningful boundaries and fine details. RCF addresses this by combining convolutional features across layers, achieving accurate and efficient BSDS500 performance.
- Edge detection extracts visually salient edges and object boundaries, supporting applications such as object detection, proposal generation, and image segmentation.
- Traditional low-level cues struggle to represent the high-level information associated with semantically meaningful edges and boundaries.
- Convolutional features become progressively coarser, while intermediate layers preserve fine details that previous architectures often ignore.
- RCF combines complementary features from all CNN convolutional layers for pixel-wise, image-to-image edge prediction across object and object-part scales.
- RCF edges have also been applied to weakly supervised semantic segmentation, style transfer, stereo matching, and other vision tasks.
- 0.811 ODS F-measure at 8 FPS is reported on BSDS500, while a fast RCF version reaches 0.806 at 30 FPS.
2 RELATED WORK
Earlier edge detectors relied on hand-crafted low-level cues or CNN features from limited layers. RCF instead combines features from all convolutional layers in a fully convolutional network.
- Early methods primarily used intensity and color gradients, while feature-learning approaches predicted edge strength from low-level intensity, gradient, and texture features.
- Deep edge detectors introduced CNN representations and approaches including nearest-neighbor search, contour subclasses, and image-to-image prediction.
- Existing CNN-based models generally used features from only the last layer of each convolutional stage rather than the full feature hierarchy.
- RCF addresses this limitation with a fully convolutional network that efficiently combines features from all convolutional layers.
3 RICHER CONVOLUTIONAL FEATURES (RCF)
RCF combines convolutional features from all VGG16 layers into a holistic, fully convolutional edge-prediction network. It preserves multiscale information, trains stage-wise and fused outputs, and uses multiscale testing to improve accuracy.
- Network Architecture: RCF removes VGG16 fully connected and pool5 layers to support image-to-image prediction while avoiding degraded edge localization.The network accepts arbitrary-sized images and outputs an edge possibility map of the same size.
- Network Architecture: Each VGG16 convolutional layer produces a 1 × 1 projection, and feature maps within each stage are accumulated into hybrid features.These hybrid features are subsequently up-sampled for prediction.
- Network Architecture: Stage-wise up-sampled features are each connected to cross-entropy and sigmoid layers, then concatenated and fused into the final output.The architecture therefore produces both intermediate-stage supervision and a fused prediction.
- Feature Representation: RCF encapsulates all convolutional features to learn multiscale information from differing receptive fields, combining coarse high-level responses with finer details.This holistic representation is trained by back-propagation and is intended for edge detection across scales.
- Annotator-robust Loss Function: The loss uses annotator-derived positive and negative samples, balancing their counts with λ while ignoring ambiguous edge pixels.The loss is computed for stage outputs and the fusion output across image pixels and network stages.
- Multiscale Hierarchical Edge Detection: Three-scale testing improves BSDS500 ODS F-measure from 0.806 to 0.811 at 8 FPS by averaging resized edge maps.The tested scales are 0.5, 1.0, and 1.5; the approach trades speed for accuracy.
4 EXPERIMENTS ON EDGE DETECTION
RCF is evaluated for edge detection on BSDS500, NYUD, and Multicue, with experiments examining accuracy, efficiency, multiscale testing, input modalities, and architecture variants.
- BSDS500 Dataset: 2.3% and 1.8% higher ODS F-measures than HED are achieved by RCF-MS and RCF, respectively, on BSDS500.Both versions also exceed the reported average human performance of 0.803 ODS F-measure.
- BSDS500 Dataset: 0.811 ODS F-measure at 8 FPS is achieved by RCF on BSDS500, while its fast version reaches 0.806 at 30 FPS.The multiscale version improves the ODS F-measure from 0.806 to 0.811, while speed decreases from 30 FPS to 8 FPS.
- NYUD Dataset: NYUD experiments train separate RGB and HHA models, encoding depth through horizontal disparity, height above ground, and angle with gravity.The RGB and HHA models are trained with the same λ setting of 1.2.
- NYUD Dataset: RCF outperforms HED on NYUD using HHA, RGB, and merged RGB-HHA data, with ODS gains of 2.2%, 2.6%, and 2.4%, respectively.Averaging HHA and RGB edges performs better than either input type alone, while HHA alone performs worse than RGB.
- NYUD Dataset: RCF with ResNet50 improves ODS F-measure by 1.6% over RCF with VGG16 on NYUD edge detection.The study also evaluates Multicue boundary and edge detection using separate training settings and random 500 × 500 crops.
- Architecture Analysis: Mixed-network experiments connect richer-feature side outputs to some VGG16 stages and HED side outputs to others, while added nonlinearities worsen performance.Adding nonlinear layers after specified 1 × 1 convolutional layers can also prevent convergence.
5 EXPERIMENTS ON IMAGE SEGMENTATION
The paper evaluates RCF edges within classical image segmentation on BSDS500 and NYUD, finding strong boundary and region results with dataset- and backbone-dependent comparisons.
- Experimental Setup: RCF segmentation replaces HED edges within the COB framework and is evaluated with VGG16 and ResNet backbones on BSDS500 and NYUD.The evaluation uses both boundary and region measures.
- BSDS500 Dataset: 1.3% higher boundary ODS and OIS F-measures than COB are achieved by RCF on BSDS500.RCF also exceeds COB by 2.4% in region ODS F-measure and 3.0% in region OIS F-measure.
- BSDS500 Dataset: RCF achieves the reported state of the art on BSDS500 segmentation for both boundary and region quality.Using ResNet as the backbone further improves performance, while boundary quality reaches human performance but region quality remains below it.
- NYUD Dataset: On NYUD, RCF with VGG16 exceeds COB on region F-measure but is slightly worse on boundary F-measure; ResNet50 yields similar boundary performance and 1.6% higher region performance.Both COB and RCF outperform traditional methods by a large margin.
- Qualitative Results: The examples display origin images, ground truth, RCF edge maps, and RCF UCM maps for BSDS500 and NYUD.The figure places BSDS500 examples in the top two rows and NYUD examples in the bottom two rows.
6 CONCLUSION
The paper presents RCF as a CNN architecture that combines hierarchical convolutional features for accurate and efficient edge detection, with competitive image-segmentation results.
- Conclusion: RCF leverages all convolutional features to combine semantic and fine-detail information in a unified edge-detection architecture.The conclusion describes the architecture as accurate, efficient, and potentially applicable to other vision tasks.
- Conclusion: RCF edges produce competitive results when applied to classical image segmentation.The paper identifies broader exploration of the architecture in other vision tasks as a future direction.