Source-linked AI summary
Cascaded Partial Decoder for Fast and Accurate Salient Object Detection
Zhe Wu, Li Su, Qingming Huang
TL;DR
Salient object detectors gain from multi-level feature aggregation, but low-level features contribute less while imposing substantial computation. CPD uses partial decoders that omit shallower features and cascaded saliency-based refinement of deeper features. On five benchmark datasets, it reports state-of-the-art performance with much faster execution and also improves existing aggregation models’ efficiency and accuracy.
Problem
Low-level features contribute less to deep aggregation performance while their larger resolutions increase computation, creating a need for efficient salient object detection.
Method
CPD discards shallower features with partial decoders and uses an initial saliency map to refine deeper features through cascaded optimization, holistic attention, and a bifurcated backbone.
Results
The model achieves state-of-the-art performance on five benchmark datasets and runs much faster than existing models, while improving existing aggregation models’ efficiency and accuracy.
Takeaways & Limitations
CPD provides a framework that can both accelerate salient object detection and improve existing multi-level feature aggregation models.
Takeaways & Limitations
Performance depends on the attention branch; classifying clutter as salient regions can produce wrong results.
Abstract
from arXiv · showhide
Existing state-of-the-art salient object detection networks rely on aggregating multi-level features of pre-trained convolutional neural networks (CNNs). Compared to high-level features, low-level features contribute less to performance but cost more computations because of their larger spatial resolutions. In this paper, we propose a novel Cascaded Partial Decoder (CPD) framework for fast and accurate salient object detection. On the one hand, the framework constructs partial decoder which discards larger resolution features of shallower layers for acceleration. On the other hand, we observe that integrating features of deeper layers obtain relatively precise saliency map. Therefore we directly utilize generated saliency map to refine the features of backbone network. This strategy efficiently suppresses distractors in the features and significantly improves their representation ability. Experiments conducted on five benchmark datasets exhibit that the proposed model not only achieves state-of-the-art performance but also runs much faster than existing models. Besides, the proposed framework is further applied to improve existing multi-level feature aggregation models and significantly improve their efficiency and accuracy.
1. Introduction
Deep salient object detectors commonly aggregate multi-level CNN features, but low-level features add computation despite quickly saturating performance. CPD addresses this trade-off by discarding shallower features and refining deeper features with generated saliency maps.
- Encoder-decoder detectors combine high-level semantic features with low-level spatial details to produce saliency maps.
- Performance saturates quickly as DSS features are progressively aggregated from high-level to low-level layers.
- VGG16 Conv3_3 retains edge information, allowing CPD to omit the larger-resolution Conv1_2 and Conv2_2 features.
- Large low-level feature resolutions increase computational complexity, although salient-object detection is often used as preprocessing for later operations.
- CPD uses a partial decoder over deeper features, then refines backbone features with an initial saliency map to improve representation ability.
- The framework combines a bifurcated backbone, cascaded optimization, and holistic attention, while supporting integration with existing aggregation models.
- Experiments on five benchmark datasets report state-of-the-art performance and much faster execution than existing models.
- Applying CPD to existing deep aggregation models significantly improves both their efficiency and accuracy.
2. Related Work
Deep salient object detection has progressed from CNN region scoring to fully convolutional and multi-level feature aggregation approaches. Related models combine semantic and spatial information through increasingly varied decoder and attention designs.
- Early CNN-based methods predicted saliency for image regions but were time-consuming.
- Fully convolutional architectures subsequently provided more effective foundations for salient object detection and related segmentation tasks.
- Multi-level feature fusion improves dense prediction by combining high-level semantic information with low-level spatial details for object-boundary refinement.
- The conventional encoder-decoder framework produces saliency map S by applying a full decoder to all feature levels.
- Prior decoders integrate features across levels using short connections, multi-resolution grids, bidirectional gating, or contextual attention.
3. The Proposed Framework
The cascaded partial decoder uses two branches: an attention branch generates an initial saliency map that refines deeper features for a detection branch. It combines deeper-layer features with holistic attention and efficient context processing while discarding shallower features to reduce computation.
- Partial Decoder: The partial decoder integrates deeper features while discarding shallower-layer features, reducing computation relative to full multi-level aggregation.The framework can also embed existing aggregation decoders in both branches, with experiments showing both branches outperform the original models.
- Attention Refinement: The attention branch aggregates three deeper features to produce an initial saliency map, enhances it with holistic attention, and uses it to refine the optimization-layer feature.The refined feature is obtained by element-wise multiplying the feature with the enhanced attention map before detection decoding.
- Cascaded Framework: The framework uses two branches with separate parameters, where the attention branch refines features used by the detection branch.The branches are jointly trained with ground truth, and the attention branch supplies an attention map while the detection branch provides an auxiliary loss.
- Holistic Attention Module: The holistic attention module enlarges the initial saliency map’s coverage so whole salient objects and their boundaries are better represented.It applies Gaussian blurring, normalization, and a maximum operation, while adding little computation.
4.1. Salient Object Detection
The proposed model is evaluated against state-of-the-art salient object detectors on five benchmark datasets and is also used to improve existing aggregation models. Results show strong performance, faster execution, and further gains in the improved models.
- The evaluation uses five benchmark datasets, MAE, max F-measure, average F-measure, and execution speed for comparison.
- The proposed model outperforms other methods in most cases and runs much faster than existing models.PiCANet-R achieves a higher maxF on DUT-OMRON, but the proposed model runs about 12 times faster.
- The proposed model recovers precise salient-object boundaries, segments small objects accurately, and produces more uniform salient regions in challenging cases.These qualitative findings are consistent with the larger improvement reported for average F-measure than for MAE and maxF.
- Application in Existing Models: The framework is applied to BMPM, Amulet, and NLDF by integrating each model’s aggregation algorithm within the two-branch framework.
- Application in Existing Models: Each improved model outperforms its original counterpart on five benchmark datasets, with larger gains on DUT-OMRON and DUTS.The improved models (-CPD and -CPD-A) run about 2 and 3 times faster than the original models, respectively.
NLDF-CPD
The NLDF-CPD comparison examines holistic attention and optimization-layer choices within the improved-model framework. Holistic attention performs better than initial attention, while the selected optimization setting balances accuracy and efficiency.
- Holistic attention outperforms initial attention in the proposed model and the three improved models.
- Selection of Optimization Layer: Refining Conv2 2 increases computational complexity and decreases performance relative to the selected optimization setting.
- Selection of Optimization Layer: Refining Conv4 3 decreases both computation cost and performance because its feature resolution is smaller.
- Selection of Optimization Layer: The Conv2 2 and Conv4 3 settings both outperform the full decoder in accuracy and efficiency.
4.2. Application in Other Tasks
The framework is evaluated beyond salient object detection on shadow detection and portrait segmentation. It outperforms existing algorithms in both applications, while the framework analysis compares alternative decoder settings and attention mechanisms.
- Shadow Detection: The proposed model is retrained and evaluated on three public shadow-detection datasets using balanced error rate.
- Framework Analysis: The framework analysis compares the proposed model with different optimization layers and with no optimization layer, using the full decoder as the latter setting.
- Framework Analysis: Holistic attention is compared with initial attention in the proposed model and three improved models.
- Shadow Detection: The proposed model outperforms the compared shadow-detection methods on all evaluated datasets.
- Portrait Segmentation: The proposed model outperforms existing algorithms on the portrait-segmentation dataset.
5. Conclusion
The paper concludes that CPD provides fast and accurate salient object detection by discarding shallow features and refining deeper features with generated saliency maps. It also improves existing aggregation models and generalizes to shadow detection and portrait segmentation.
- CPD discards shallower decoder features for efficiency and uses generated saliency maps to refine backbone features for accuracy.
- The model achieves state-of-the-art performance on five benchmark datasets while running much faster than existing deep models.
- Applying the framework to existing deep aggregation models significantly improves their accuracy and efficiency.
- The proposed model is also validated on shadow detection and portrait segmentation.