Source-linked AI summary
Pyramid Feature Attention Network for Saliency detection
Ting Zhao, Xiangqian Wu
TL;DR
Saliency detection requires effective feature extraction because indiscriminate multi-scale fusion can include interfering features. PFA addresses this by processing high-level context and low-level spatial features with CPFE, channel-wise and spatial attention, and edge-preserving loss. The method is reported to achieve state-of-the-art performance on several challenging datasets.
Problem
Existing saliency methods integrate multi-scale convolutional features without considering their differing contributions, although high-level and low-level features serve different roles.
Method
PFA combines context-aware pyramid feature extraction and channel-wise attention for high-level features with spatial attention for low-level features, plus edge preservation loss.
Results
The proposed model achieves state-of-the-art performance on several challenging datasets.
Takeaways & Limitations
PFA focuses feature processing on context-rich high-level representations, spatially detailed low-level representations, and boundary information.
Takeaways & Limitations
Hand-crafted features and priors can hardly capture high-level and global semantic knowledge about objects.
Abstract
from arXiv · showhide
Saliency detection is one of the basic challenges in computer vision. How to extract effective features is a critical point for saliency detection. Recent methods mainly adopt integrating multi-scale convolutional features indiscriminately. However, not all features are useful for saliency detection and some even cause interferences. To solve this problem, we propose Pyramid Feature Attention network to focus on effective high-level context features and low-level spatial structural features. First, we design Context-aware Pyramid Feature Extraction (CPFE) module for multi-scale high-level feature maps to capture rich context features. Second, we adopt channel-wise attention (CA) after CPFE feature maps and spatial attention (SA) after low-level feature maps, then fuse outputs of CA & SA together. Finally, we propose an edge preservation loss to guide network to learn more detailed information in boundary localization. Extensive evaluations on five benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art approaches under different evaluation metrics.
1. Introduction
Saliency detection needs features that capture high-level semantic context while preserving low-level object boundaries. The PFA network addresses indiscriminate multi-scale fusion by applying attention to different feature levels and adding edge-preserving supervision.
- Fully convolutional networks provide high-level semantic information, but pooling reduces feature-map resolution and deteriorates salient-object boundaries.
- Separately extracted hand-crafted and CNN features are complementary, but their fusion is difficult and hand-crafted extraction is time-consuming.
- Deep features capture global context for locating salient regions, whereas shallow features preserve spatial structure for boundary localization.
- PFA uses CPFE and channel-wise attention on high-level features to select useful scales and receptive fields for salient-region prediction.The attention assigns larger weights to channels that respond strongly to salient objects.
- PFA applies spatial attention to low-level features to suppress background details and focus on effective boundaries.
- A novel edge preservation loss guides the network to learn more detailed information for boundary localization.
- The proposed model achieves state-of-the-art performance on several challenging datasets, with experiments reported as demonstrating its effectiveness and superiority.
2. Related Works
Prior saliency methods use hand-crafted priors or multi-level CNN features, but each approach has limitations in representing semantics or distinguishing feature contributions. PFA combines context-aware extraction and attention mechanisms to select high- and low-level information differently.
- Hand-crafted saliency features preserve fine image structure but have difficulty capturing high-level and global semantic knowledge about objects.
- Deep CNN layers encode abstract semantic information, while shallow layers retain spatial details for reconstructing object boundaries.
- Attention mechanisms are used across tasks including translation, recognition, captioning, visual question answering, and pose estimation.
- PFA uses multi-scale atrous convolutions and channel-wise attention to capture context-rich high-level features rather than treating multi-level features uniformly.
- The architecture uses CPFE on high-level feature maps and combines low-level feature maps after upsampling them to the vgg1-2 size.
- Spatial attention filters background details in low-level features according to high-level features, emphasizing foreground regions for saliency prediction.
3. Pyramid Feature Attention Network
PFA separates high-level context modeling from low-level spatial refinement, using attention to select effective features and an edge-aware loss to improve boundary localization.
- Network overview: PFA combines context-aware pyramid extraction, channel-wise attention, spatial attention, and edge preservation loss for saliency detection.High-level features are processed for context, low-level features for spatial detail, and the resulting objectives include boundary supervision.
- Context-aware pyramid feature extraction: Atrous convolutions with dilation rates 3, 5, and 7 capture multi-receptive-field context from VGG-16 high-level features.Features from conv3-3, conv4-3, and conv5-3 are combined with a 1×1 dimension-reduction feature by cross-channel concatenation.
- Channel-wise attention: Channel-wise attention weights high-level channels according to their responses to salient objects.The attention representation uses average pooling, two fully connected layers, and sigmoid normalization before weighting the pyramid features.
- Spatial attention: Spatial attention focuses low-level features on foreground regions instead of treating all spatial positions equally.Two asymmetric convolution layers, with kernels 1×k and k×1, expand the receptive field before sigmoid normalization; the experiment sets k=9.
- Edge preservation loss: The loss combines cross-entropy saliency supervision with Laplace-derived boundary supervision to emphasize salient-object boundaries.The Laplace operator extracts boundaries from the ground truth and network saliency map, after which cross-entropy supervises boundary generation.
4. Experiments
Experiments evaluate the proposed method on five benchmark datasets using standard saliency metrics and comparisons with eleven CNN-based approaches. Results indicate stronger quantitative and visual performance, while ablations support the contributions of attention modules and edge preservation loss.
- Datasets and Evaluation Criteria: Evaluation uses DUTS-test, ECSSD, HKU-IS, PASCAL-S, and DUT-OMRON with PR curves, F-measure, wFβ, and MAE.The implementation uses VGG-16 pretrained on ImageNet and trains on DUTS-train with augmentation and a two-stage learning setup.
- Quantitative Comparison: The proposed method is compared with eleven state-of-the-art salient object detection approaches using author-provided saliency maps or recommended implementations.The comparison includes BDMPM, GRL, PAGRN, Amulet, SRM, UCF, DCL, DHS, ELD, NLDF, and RFCN.
- Visual Comparison: Visual comparisons show clearer salient regions, stronger background suppression, and sharper boundaries than other approaches.The reported advantages include cases where salient objects resemble the background or contain special semantic information.
- Quantitative Comparison: The method achieves the best wFβ and MAE results across all five test datasets, while its PR and F-measure curves are higher than competing methods.The authors report a larger improvement over the best existing approach on the challenging DUT-OMRON dataset.
- Edge Preservation Loss: Edge preservation loss improves boundary detail, and α = 0.7 gives the best result in the reported experiment.Figure 7 compares outputs with and without the loss, while Table 2 evaluates different α settings on DUTS-test.
- Ablation Study: Ablation results show cumulative MAE reductions of 37% after adding channel attention, 57% after spatial attention, and 60% after edge preservation loss.The baseline using only high-level features has MAE 0.1003; the full model performs best among the tested component combinations.
5. Conclusions
The paper concludes that Pyramid Feature Attention combines context-aware high-level feature extraction, channel attention, spatial attention, and edge preservation loss for saliency detection. Experiments on five datasets report performance exceeding state-of-the-art methods under different evaluation metrics.
- Conclusions: Pyramid Feature Attention uses multi-scale atrous convolutions and channel attention to capture semantic high-level features.Spatial attention is applied to low-level features to suppress background noise and focus on salient objects.
- Conclusions: Edge preservation loss guides the network toward more detailed boundary localization.The loss complements feature processing by targeting boundary detail explicitly.
- Conclusions: Experiments on five datasets report that the approach outperforms state-of-the-art methods under different evaluation metrics.The conclusion characterizes the proposed network as effective for saliency detection within the evaluated setting.