Source-linked AI summary
Amulet: Aggregating Multi-level Convolutional Features for Salient Object Detection
Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, Xiang Ruan
TL;DR
Salient object detection lacks a satisfactory way to aggregate multi-level convolutional features while retaining both semantic information and fine boundaries. Amulet integrates and adaptively combines features at multiple resolutions, recursively embedding edge-aware maps and low-resolution predictions. Trained only on MSRA10K, it achieves new state-of-the-art performance on seven other datasets and runs at 16 fps.
Problem
Salient object detection needs better aggregation of multi-level convolutional feature maps because existing approaches do not adequately combine the relevant cues.
Method
Amulet integrates multi-level FCN features at multiple resolutions, learns to combine them, and recursively embeds edge-aware maps and low-resolution predictions.
Results
The model, trained only on MSRA10K, achieves new state-of-the-art performance on seven other salient object detection datasets and runs at 16 fps.
Takeaways & Limitations
Aggregating multi-level features with edge-aware and recursive information produces accurate salient object labeling that performs favorably against state-of-the-art methods.
Abstract
from arXiv · showhide
Fully convolutional neural networks (FCNs) have shown outstanding performance in many dense labeling problems. One key pillar of these successes is mining relevant information from features in convolutional layers. However, how to better aggregate multi-level convolutional feature maps for salient object detection is underexplored. In this work, we present Amulet, a generic aggregating multi-level convolutional feature framework for salient object detection. Our framework first integrates multi-level feature maps into multiple resolutions, which simultaneously incorporate coarse semantics and fine details. Then it adaptively learns to combine these feature maps at each resolution and predict saliency maps with the combined features. Finally, the predicted results are efficiently fused to generate the final saliency map. In addition, to achieve accurate boundary inference and semantic enhancement, edge-aware feature maps in low-level layers and the predicted results of low resolution features are recursively embedded into the learning framework. By aggregating multi-level convolutional features in this efficient and flexible manner, the proposed saliency model provides accurate salient object labeling. Comprehensive experiments demonstrate that our method performs favorably against state-of-the art approaches in terms of near all compared evaluation metrics.
1. Introduction
Salient object detection remains difficult because visual saliency depends on many factors, while existing FCN approaches do not fully resolve multi-level feature aggregation and boundary preservation. Amulet addresses these issues by recursively aggregating multi-level features and edge-aware information, achieving state-of-the-art performance across multiple datasets.
- Salient object detection remains unsolved because many visual factors contribute to saliency and are difficult to combine appropriately.
- Existing FCN-based saliency models use high-level semantic features but mainly focus on non-linear combinations of high-level information.
- AmuletNet integrates convolutional features from multiple levels into multiple resolutions, learns their combinations, and recursively predicts saliency maps.
- Deep recursive supervision incorporates low-level edge-aware maps and low-resolution predictions to improve boundary inference and semantic enhancement without additional annotations.
- The model, trained only on MSRA10K, achieves new state-of-the-art performance on seven other salient object detection datasets and runs at 16 fps.
2. Related Work
Earlier saliency methods rely on hand-crafted low-level cues, while deep models improve semantics but often lose spatial detail. Amulet jointly aggregates multi-level features at multiple resolutions, uses bidirectional information flow, and refines semantic predictions with low-level boundary cues.
- Traditional salient object detection methods predominantly use hand-crafted contrast, color, and texture features.
- Deep CNN approaches deliver stronger saliency performance, yet several models imperfectly handle low-level details and use computationally expensive fully connected layers.
- Convolutional features at different levels provide complementary information: high-level features support category recognition, whereas low-level features produce sharp boundaries.
- Amulet’s distinctions: Amulet jointly aggregates multi-level features at multiple resolutions, unlike prior stagewise approaches, incorporating coarse semantics and fine details together.
- Amulet’s distinctions: Its bidirectional information stream complements prediction, while low-level edge-aware maps refine high-level semantic predictions and preserve object boundaries.
3. Aggregating Convolutional Feature Model
AmuletNet aggregates convolutional features across levels and resolutions, then recursively predicts and refines saliency maps. Its architecture combines coarse semantics with fine details, propagates supervision bidirectionally, and preserves object boundaries.
- Architecture: AmuletNet jointly trains multi-level feature extraction, resolution-based feature integration, recursive saliency prediction, and boundary-preserved refinement.The final saliency map is produced by fusing multiple predicted saliency maps.
- Multi-level feature extraction: VGG-16 supplies feature maps from multiple convolutional levels for subsequent integration.The architecture removes the last pooling stage, producing feature maps reduced by a factor of 16 relative to the input.
- Resolution-based feature integration: RFCs resize multi-level features to common resolutions, concatenate them, and use 1×1 convolution to weight and combine the channels.For AmuletNet, five levels are integrated into five resolutions, with 320 concatenated channels reduced to 64 integrated feature maps.
- Recursive saliency map prediction: Recursive saliency prediction uses integrated features together with higher-level predictions to connect predictions across feature levels.The prediction equation combines deconvolved integrated features, higher-level predictions, bias, activation, and recursive weights.
- Bidirectional information aggregating learning: Deep recursive supervision propagates pixel-wise ground-truth guidance across levels and creates bidirectional information exchange between predictions.The framework is trained end-to-end using single-resolution ground truth without additional annotations.
- Boundary-preserved refinement: Boundary-preserved refinement uses edge-aware low-level conv1-2 features to refine predicted object boundaries before final saliency fusion.A learned convolutional layer produces the final fused saliency prediction from the boundary-refined results.
4. Experiments
Experiments evaluate Amulet on seven public saliency datasets using standard metrics, comparisons with 11 state-of-the-art methods, qualitative examples, and ablations of feature resolution and boundary refinement. The model generally achieves strong quantitative and qualitative performance, while multi-level features and BPRs improve results, especially boundary accuracy.
- Experimental Setup: Amulet is evaluated on seven public saliency datasets using PR curves, F-measure, and MAE.The comparison includes six large-scale datasets in Table 1 and the broader seven-dataset evaluation described in the experimental setup.
- Performance Comparison: The comparison covers 11 state-of-the-art methods, including seven deep-learning and four conventional saliency detectors.Implementations use recommended parameters or saliency maps supplied by the authors for fair comparison.
- Performance Comparison: Amulet largely outperforms compared methods across datasets and nearly all evaluation metrics, with higher F-measure and generally lower MAE.The reported F-measure gains are especially notable on DUTS-TE, ECSSD, and HKU-IS.
- Qualitative Evaluation: Qualitative comparisons show more accurate saliency maps for low-contrast objects, boundary-adjacent objects, and multiple disconnected salient objects.The BPR component further improves salient-object boundaries in the illustrated cases.
- Ablation Studies: Using higher-resolution integrated features progressively improves performance, while even Amulet-1/16 outperforms most existing methods.The resolution study reports that features from all levels are helpful for saliency detection.
- Ablation Studies: Removing BPRs causes a modest F-measure decrease but a large MAE increase, indicating their role in detecting and localizing salient-object boundaries.Visual examples in Figure 5 compare Amulet predictions with and without BPRs.
5. Conclusion
Amulet is a generic framework for salient object detection that aggregates multi-level convolutional features across multiple resolutions. It also embeds edge-aware maps and high-level predictions, and performs favorably against state-of-the-art approaches.
- Amulet integrates multi-level feature maps into multiple resolutions, learns to combine them, and predicts saliency maps with the integrated features.
- Edge-aware maps and high-level predictions are embedded into the framework.
- Experiments show that Amulet performs favorably against state-of-the-art approaches in saliency detection.