Source-linked AI summary
Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform
Liang-Chieh Chen, Jonathan T. Barron, George Papandreou, Kevin Murphy, Alan L. Yuille
TL;DR
Semantic segmentation benefits from boundary-aware CRF post-processing, but dense CRF inference is computationally expensive. The paper replaces it with recursive domain transform filtering and learns task-specific edges from CNN features; the resulting system is several times faster while producing almost comparable segmentation results and competitive edge maps.
Problem
Fully connected CRFs improve semantic segmentation near object boundaries, but their bilateral filtering requires substantial memory and CPU time.
Method
The method replaces dense CRF processing with recursive domain transform filtering and learns its reference edge map from intermediate features of the segmentation CNN.
Results
The system is several times faster than fully connected CRFs while performing almost as well on PASCAL VOC 2012 mIOU and competitively on BSDS500 edge detection.
Takeaways & Limitations
End-to-end task-specific edge learning improves localization near object boundaries while retaining an efficient filtering scheme.
Abstract
from arXiv · showhide
Deep convolutional neural networks (CNNs) are the backbone of state-of-art semantic image segmentation systems. Recent work has shown that complementing CNNs with fully-connected conditional random fields (CRFs) can significantly enhance their object localization accuracy, yet dense CRF inference is computationally expensive. We propose replacing the fully-connected CRF with domain transform (DT), a modern edge-preserving filtering method in which the amount of smoothing is controlled by a reference edge map. Domain transform filtering is several times faster than dense CRF inference and we show that it yields comparable semantic segmentation results, accurately capturing object boundaries. Importantly, our formulation allows learning the reference edge map from intermediate CNN features instead of using the image gradient magnitude as in standard DT filtering. This produces task-specific edges in an end-to-end trainable system optimizing the target semantic segmentation quality.
1. Introduction
The paper replaces computationally expensive fully connected CRF processing with domain transform filtering and learns task-specific reference edges within a unified CNN. The method is substantially faster while achieving nearly comparable segmentation quality and competitive edge detection.
- Motivation: Fully connected CRF post-processing improves segmentation near object boundaries, but its bilateral filtering is expensive in memory and CPU time.The filtering operates as high-dimensional Gaussian filtering in a 5-D bilateral space.
- Approach: The proposed method replaces the fully connected CRF and bilateral filtering with domain transform filtering, an efficient alternative edge-aware filter.Its recursive formulation adaptively filters signals without propagating information across reference edges.
- Connection to recurrent networks: The domain transform can equivalently be understood as a recurrent neural network and as a special case of gated recurrent units.This connection provides shared insights into the two methods.
- Approach: The system learns the reference edge map from intermediate CNN features rather than using standard image gradient magnitude.This creates a task-specific edge detector optimized jointly with semantic segmentation in an end-to-end trainable system.
- Evaluation: On PASCAL VOC 2012, domain transform filtering is several times faster than dense CRF inference while performing almost as well on mean intersection-over-union.The learned edge map also performs competitively on the BSDS500 edge detection benchmark.
2. Related Work
The related work connects this approach to CNN-based boundary estimation, learned edge detection, semantic segmentation, and recurrent models for spatial dependencies.
- Semantic image segmentation: CNN segmentation systems use intermediate features, deconvolution, and unpooling to improve object-boundary localization after downsampling and max-pooling.These approaches address spatial detail lost in the segmentation output.
- Edge detection: Prior CNN edge detectors learned boundaries from intermediate deep-network features but did not apply those edges to high-level tasks such as semantic segmentation.Other cited methods directly optimized learned boundaries for semantic segmentation.
- Long range dependency: The paper relates CNN-DT integration to recurrent approaches that model long-range spatial dependencies in computer vision.The domain transform uses recursive filtering within this broader recurrent-modeling context.
3. Proposed Model
The proposed model combines DeepLab segmentation, EdgeNet edge prediction, and domain-transform filtering in an end-to-end trainable architecture. The domain transform filters coarse scores using learned reference edges, and its recursive computation is differentiable so segmentation errors can train both inputs.
- Model overview: The model jointly trains DeepLab, EdgeNet, and Domain Transform to optimize final semantic segmentation quality.DeepLab produces coarse scores, EdgeNet predicts edges from intermediate DeepLab features, and Domain Transform filters the scores using those edges.
- Model overview: EdgeNet resizes and concatenates intermediate DeepLab features before a 1×1 convolution produces nonnegative edge predictions.A ReLU constrains the edge prediction to zero or greater.
- Domain transform filtering: Domain Transform filters coarse CNN segmentation scores with separable one-dimensional recursive passes across rows and columns, guided by an edge map.The two-dimensional procedure applies bidirectional filtering horizontally and then vertically.
- Domain transform filtering: The domain-transform weight acts as a diffusion gate: small density enables propagation, whereas large density stops diffusion and preserves the raw input.When density is very small, wi = 1 and yi = yi−1; when it is very large, wi = 0 and yi = xi.
- Trainable filtering: Unlike standard DT filtering based on image gradients, the model learns task-specific reference edges from CNN features and backpropagates segmentation errors through the filter.This makes the DT a trainable CNN layer that jointly learns coarse scores and reference edges.
- Relation to gated recurrent units: The recursive DT operation has a precise correspondence with a GRU, relating the DT weight to the GRU update gate and the raw input to candidate activation.The correspondence is zi = 1 − wi and ỹi = xi.
4. Experimental Evaluation
Experiments evaluate the proposed domain-transform models on PASCAL VOC 2012, including hyper-parameter effects, edge detectors, boundary accuracy, and comparisons with dense CRF inference. The learned-edge approach improves segmentation over DeepLab while approaching dense CRF accuracy at substantially lower computation.
- Experimental Protocol: Performance is measured on PASCAL VOC 2012 across 20 foreground classes and one background class using mean pixel IOU.The metric averages intersection-over-union across all 21 classes.
- Hyper-parameters: K = 3 domain-transform iterations recover most gains across several σs and σr settings while reducing striping artifacts from one-dimensional filtering.The filtering process uses separable row and column passes, and multiple iterations suppress striping.
- Hyper-parameters: Increasing σr can introduce background textures into learned edges, whereas σs changes the edges little once σs ≥90.Noisy edges make propagation between neighboring pixels harder for semantic segmentation.
- Validation Results: DT-Gradient improves DeepLab by 1.7%, DT-SE adds 0.9% over DT-Gradient, and DT-EdgeNet reaches 4.1% above baseline on the validation set.DT-EdgeNet remains 1.2% below DeepLab-CRF but is several times faster; combining it with dense CRF improves performance by 0.8% over DeepLab-CRF.
- Test Results: On the test set, DT-EdgeNet reaches 69.0%, 3.9% above baseline and 1.3% below DeepLab-CRF; adding dense CRF raises it to 71.2%.With MS-COCO pretraining, DT-EdgeNet improves DeepLab by 2.8%, is 1.0% below DeepLab-CRF, and reaches 73.6% when combined with dense CRF.
- Additional Evaluation: The method reaches 76.3% on the test set with multi-scale inputs, 1.5% behind current best models, and its learned edges achieve ODS=0.718, OIS=0.731, and AP=0.685 on BSDS500.The edge detector was trained only for semantic segmentation and had no BSDS500 edge supervision.
- Efficiency: Domain transform takes 180 ms/image on CPU versus 830 ms/image for dense CRF, while GPU DT inference takes 25 ms/image and EdgeNet+DT adds about 18% overhead to DeepLab.The timing uses K = 3 DT iterations on 50 VOC validation images.
5. Conclusions
The paper presents an end-to-end system that learns task-specific edge maps for semantic segmentation using a domain transform. The resulting filtering improves boundary localization while offering faster inference than fully connected CRFs.
- The method learns edge maps useful for semantic image segmentation in a unified, discriminatively trained end-to-end system.
- Backpropagation through the domain transform learns task-specific edges optimized for semantic segmentation.
- The learned domain transform improves localization accuracy near object boundaries compared with raw semantic segmentation maps.
- Table 5 reports average inference time in milliseconds per image, with percentages relative to DeepLab computation.
Appendix
The appendix provides detailed quantitative and qualitative supplementary results for the proposed methods.
- The appendix reports per-class semantic segmentation IOU on PASCAL VOC 2012 and additional edge detection and segmentation examples.
A. Detailed quantitative image segmentation results
This appendix section reports per-class segmentation performance on the PASCAL VOC 2012 test set under two pretraining settings.
- Per-class semantic segmentation IOU is compared with DeepLab-LargeFOV and DeepLab-CRF-LargeFOV baselines.
- Models are evaluated after ImageNet-only pretraining or additional MS-COCO semantic segmentation pretraining.
B. Qualitative edge detection and image segmentation results
The qualitative appendix compares domain-transform segmentation using learned EdgeNet edges with segmentation using Structured Edges on PASCAL VOC 2012 validation images.
- B. Qualitative edge detection and image segmentation results: Additional PASCAL VOC 2012 validation examples compare domain-transform results using learned EdgeNet edges against SE edges.
- B. Qualitative edge detection and image segmentation results: Figures 10 and 11 show images, baseline segmentation, edge maps, and segmentation results for Structured Edges and EdgeNet.