Source-linked AI summary
Recurrent Attentional Networks for Saliency Detection
Jason Kuen, Zhenhua Wang, Gang Wang
TL;DR
Saliency detection remains difficult for convolutional-deconvolutional networks when objects appear at multiple scales. The paper proposes RACDNN, which iteratively attends to image sub-regions and uses recurrent context for saliency refinement; experiments report better results than baselines and state-of-the-art methods. RACDNN's performance may nevertheless depend on the quality of its initial saliency maps.
Problem
CNN-DecNNs perform poorly on multiple-scale objects and capture long-distance context weakly because of fixed-size receptive fields and local convolution filters.
Method
RACDNN combines spatial attention with recurrent convolutional units to refine CNN-DecNN saliency maps over flexibly sized sub-regions while passing contextual information across iterations.
Results
On all five datasets and both F-measure and MAE, RACDNN outperforms the baseline methods; on HKUIS and ECSSD, its F-measure improvement over DRFI exceeds 5%.
Takeaways & Limitations
RACDNN provides a recurrent-attention approach for improving CNN-DecNN saliency maps and is reported to outperform state-of-the-art saliency detection methods.
Takeaways & Limitations
Performance may be limited by the quality of the initial saliency maps.
Abstract
from arXiv · showhide
Convolutional-deconvolution networks can be adopted to perform end-to-end saliency detection. But, they do not work well with objects of multiple scales. To overcome such a limitation, in this work, we propose a recurrent attentional convolutional-deconvolution network (RACDNN). Using spatial transformer and recurrent network units, RACDNN is able to iteratively attend to selected image sub-regions to perform saliency refinement progressively. Besides tackling the scale problem, RACDNN can also learn context-aware features from past iterations to enhance saliency refinement in future iterations. Experiments on several challenging saliency detection datasets validate the effectiveness of RACDNN, and show that RACDNN outperforms state-of-the-art saliency detection methods.
1. Introduction
The paper adapts convolutional-deconvolutional networks for end-to-end saliency detection and proposes recurrent attention to address multiple object scales. RACDNN progressively refines saliency maps using flexible sub-regions and prior contextual information, with experiments reporting improved performance.
- Saliency detection identifies salient objects and segments their boundaries, supporting applications including object recognition, visual tracking, and image retrieval.
- Conventional CNNs lose detailed information through repeated downsizing, limiting dense segmentation of salient objects.
- CNN-DecNN produces dense pixel-precise saliency maps but works poorly for multiple-scale objects because of fixed-size receptive fields.
- RACDNN uses spatial transformers and recurrent units to attend iteratively to flexibly sized image sub-regions and refine saliency predictions.
- Previous attended sub-regions provide contextual information for refining later regions, such as using visible front legs to refine less-visible back legs.
- Experiments on several challenging benchmark datasets compare RACDNN with state-of-the-art methods and report its effectiveness.
2. Related work
Prior saliency methods include bottom-up and top-down approaches, while recent deep learning methods learn features from minimally processed images. The paper distinguishes its approach by combining recurrent attention with deep learning for saliency detection.
- Bottom-up methods use local visual cues such as color, contrast, orientation, and texture.
- Top-down methods rely on high-level, task-specific prior knowledge.
- Deep learning-based methods learn low-level features and high-level semantics directly from minimally processed images instead of manually tuned saliency-specific features.
- The paper states that earlier deep learning saliency methods used neither attention mechanisms nor recurrent neural networks to improve detection.
- Attention models are often combined with recurrent neural networks to achieve sequential visual attention.
3. Proposed Method
RACDNN combines CNN-DecNN saliency prediction with spatial-transformer attention and recurrent processing to refine saliency progressively on selected image sub-regions. Its design addresses fixed receptive-field limitations by supporting flexible attention and contextual information from previous iterations.
- 3. Proposed Method: The method first generates an initial saliency map from the full image, then performs progressive local refinement on selected sub-regions.Initial prediction uses CNN-DecNN, while refinement uses another CNN-DecNN under a recurrent attentional framework.
- 3.1. Deconvolutional Networks for Salient Object Detection: CNN-DecNN produces dense pixelwise saliency maps through convolution, unpooling, and sigmoid conversion of raw outputs.The framework transforms the image into a compact representation, decodes it into a raw saliency map, and applies an element-wise sigmoid.
- 3.1. Deconvolutional Networks for Salient Object Detection: Fixed-size receptive fields make CNN-DecNN perform poorly on multiple-scale objects and limit long-distance contextual modeling.The paper motivates recurrent attention as a response to both limitations.
- 3.2. Attentional Inputs and Outputs with Spatial Transformer: Spatial transformers map output coordinates to sampling coordinates in input feature maps, enabling attention to regions with flexible spatial sizes and locations.The transformation matrix is produced by a localization network and supports spatial attention while preserving channel count.
- 3.2. Attentional Inputs and Outputs with Spatial Transformer: An inverse spatial transformer maps refined saliency outputs back to the same image sub-region attended at the input end.It uses the inverse of the input transformation matrix as the output transformation matrix.
- 3.3. Recurrent Attentional Networks for Saliency Refinement: RACDNN combines recurrent CNN processing with spatial attention so hidden states carry contextual information from earlier attended regions into later saliency refinement.The architecture uses shared recurrent CNN parameters, convolutional hidden connections, and a second recurrent layer for locational dependencies.
4. Implementation Details
RACDNN combines independently trained initial and recurrent CNN-DecNN components with recurrent iterations, specific initialization and optimization settings, and mean shift post-processing.
- Network initialization: The initial CNN uses convolutional and pooling layers initialized from VGG-CNN-S weights, while fully connected layers are discarded.VGG-CNN-S was pretrained on ImageNet and contains five convolutional and three fully connected layers.
- Evaluation: Table 1 reports F-measure scores and Mean Absolute Errors for comparisons with baseline methods.The table caption identifies the two metrics and baseline comparison.
- Optimization: Gradients are clipped to [−5, 5], and batch normalization is applied to weight layers except recurrent hidden-to-hidden connections.These choices are used during training to mitigate gradient explosion and improve convergence.
- Post-processing: Mean shift-based segmentation is applied to RACDNN outputs as post-processing to produce consistent saliency values and sharp segment edges.The post-processing step is intended to achieve effects similar to object segmentation methods.
5. Saliency Training Datasets
Because most saliency datasets are small, the paper follows a prior dataset procedure rather than independently splitting each dataset into training, testing, and validation sets.
- Dataset constraints: Small saliency detection datasets make it difficult to train deep models well using separate training, testing, and validation splits for each dataset.The paper states that learning-based methods need many training samples to generalize well.
6. Experiments
Experiments evaluate RACDNN across challenging saliency datasets using precision-recall curves, F-measure, and MAE. RACDNN outperforms baseline and state-of-the-art methods, with especially notable gains on complex scenes, while qualitative results reveal a background-like appearance limitation.
- Datasets and Evaluation Metrics: The evaluation uses precision-recall curves, weighted F-measure with β^2=0.3, and Mean Absolute Error (MAE).F-measure summarizes thresholded precision-recall pairs, while MAE compares real-valued saliency maps directly with groundtruth.
- Comparison with Baseline Methods: RACDNN achieves better F-measure and MAE results than both CNN-DecNN and the non-recurrent attentional baseline across all five datasets.The comparison isolates the contribution of recurrent attention against non-attentional and non-recurrent alternatives.
- Comparison with State-of-the-art Methods: RACDNN performs better overall than the evaluated state-of-the-art methods, with more noticeable gains on the complex-scene datasets ECSSD and HKUIS.The comparison uses precision-recall curves generated from the proposed method’s saliency maps.
- Comparison with State-of-the-art Methods: On HKUIS and ECSSD, RACDNN improves F-measure over DRFI by more than 5% and lowers MAE by a large margin.These comparisons are reported in Table 2 against state-of-the-art methods.
- Comparison with State-of-the-art Methods: Qualitatively, RACDNN better detects multiple intermingled salient objects, but tends to miss objects composed mostly of background-like colors and textures.The reported examples include a dog and rabbit, as well as sky- and soil-colored objects.
- Comparison with State-of-the-art Methods: RACDNN is comparable to MCDL and MDF in F-measure while outperforming them in MAE on HKUIS, ECSSD, and SED2.The comparison uses trained models provided by the respective authors.
7. Conclusion
The paper introduces recurrent attention combined with a convolutional-deconvolutional network for saliency detection and reports experimental effectiveness. Its performance may be limited by the quality of the initial saliency maps, motivating future end-to-end detection from scratch.
- The paper introduces recurrent attention with a convolutional-deconvolutional network to address saliency detection.
- Experiments show that the proposed method is effective for saliency detection.
- Performance may be limited by the quality of the initial saliency maps.The authors suggest revamping the recurrent attentional network to detect saliency from scratch end-to-end.