Source-linked AI summary
Saliency Detection for Stereoscopic Images Based on Depth Confidence Analysis and Multiple Cues Fusion
Runmin Cong, Jianjun Lei, Changqing Zhang, Qingming Huang, Xiaochun Cao, Chunping Hou
TL;DR
Existing saliency models largely underuse depth information in stereoscopic images. The paper introduces depth-confidence analysis, graph-based color-depth compactness, and depth-refined foreground contrast, then combines the resulting maps. On two public stereo datasets, it reports better performance than 10 state-of-the-art approaches.
Problem
Existing saliency detection models mainly focus on RGB information and do not adequately explore depth information in stereoscopic images.
Method
The method evaluates depth reliability, incorporates color and depth into graph-based compactness saliency, selects foreground seeds using depth refinement, and fuses multiple cues.
Results
The method performs better than 10 state-of-the-art approaches on two publicly available stereo datasets.
Takeaways & Limitations
Depth confidence analysis and two-stage saliency computation support the proposed stereo saliency model's reported benchmark performance.
Abstract
from arXiv · showhide
Stereoscopic perception is an important part of human visual system that allows the brain to perceive depth. However, depth information has not been well explored in existing saliency detection models. In this letter, a novel saliency detection method for stereoscopic images is proposed. Firstly, we propose a measure to evaluate the reliability of depth map, and use it to reduce the influence of poor depth map on saliency detection. Then, the input image is represented as a graph, and the depth information is introduced into graph construction. After that, a new definition of compactness using color and depth cues is put forward to compute the compactness saliency map. In order to compensate the detection errors of compactness saliency when the salient regions have similar appearances with background, foreground saliency map is calculated based on depth-refined foreground seeds selection mechanism and multiple cues contrast. Finally, these two saliency maps are integrated into a final saliency map through weighted-sum method according to their importance. Experiments on two publicly available stereo datasets demonstrate that the proposed method performs better than other 10 state-of-the-art approaches.
I. INTRODUCTION
The paper targets bottom-up saliency detection for stereoscopic images, addressing the limited use of depth information in predominantly RGB-based models. It proposes depth confidence analysis and multiple cue fusion to improve stereo saliency detection.
- Most prior saliency detection methods focus on 2D images and mainly use RGB information, while ignoring depth or disparity cues.
- Depth information can provide a useful cue for saliency detection in stereoscopic images.
- The proposed method evaluates depth-map reliability and reduces the influence of poor depth maps on saliency detection.
- It introduces compactness saliency using color and depth cues.
- Its foreground saliency stage uses depth-refined seed selection and contrast across color, depth, and texture cues.
II. PROPOSED METHOD
The proposed pipeline assesses depth-map reliability, constructs saliency representations using image regions and depth, and combines compactness and foreground saliency. Depth confidence is based on observable depth-distribution characteristics.
- The method first computes depth confidence to evaluate depth-map reliability and reduce poor-depth influence in later saliency processing.
- RGB images are segmented into superpixels and represented as graphs before compactness saliency is computed from color and depth cues.
- The final saliency map is obtained by weighted-summing compactness and foreground saliency maps.
- A. Depth Confidence Measure: Good, common, and poor depth maps are distinguished using depth-distribution characteristics including mean value, coefficient of variation, and entropy.
- A. Depth Confidence Measure: The depth-confidence measure assigns larger λd values to more reliable input depth maps.
B. Graph Construction
The graph represents SLIC superpixels as nodes and adjacency as links, with edge-related differences incorporating both CIE Lab color and depth information.
- SLIC segments the RGB image into homogeneous, compact regions, with N set to 200 in the experiments.
- The graph G = (V, E) uses superpixels as nodes and links between adjacent superpixels as edges.
- Color distance is measured as Euclidean distance in CIE Lab space, while depth difference compares mean depths between superpixels.
- Similarity between adjacent superpixels is used to define the affinity matrix W = [wij]N×N.
C. Compactness Saliency Using Color and Depth Cues
The method defines compactness saliency by combining color and depth cues, while incorporating objectness to account for superpixel location.
- Color and depth-based compactness replaces color-only compactness to compute a compactness saliency map.The formulation is motivated by compact salient regions and centralized depth distributions.
- The compactness map combines normalized color and depth compactness with an objectness measure.Objectness evaluates the probability that superpixel vi belongs to an object.
- Superpixel size weights the spatial mean used in compactness computation, emphasizing larger regions.The weighting uses nj, the number of pixels in superpixel vj.
D. Foreground Saliency Using Multiple Cues Contrast
A depth-refined foreground-seed mechanism and multiple-cue contrast address compactness errors when salient regions resemble the background.
- Compactness saliency can wrongly detect regions when salient areas have appearances similar to the background.The foreground branch is introduced to mitigate this limitation.
- DRSS selects foreground seeds using both compactness saliency and depth, rather than preliminary saliency alone.Preliminary seeds are thresholded at τ = 0.5, then refined using their mean depth.
- The foreground saliency contrasts each superpixel with the seeds using color, depth, texture, and position cues.Superpixels more similar to the foreground seeds are assigned higher foreground saliency.
- Manifold ranking propagates foreground saliency to avoid highlighting only object boundaries.The propagated map is normalized to [0, 1] to obtain the final foreground saliency map.
E. Saliency Map Integration
The complementary compactness and foreground maps are combined through a weighted sum, with γ controlling their relative contributions.
- The final saliency map integrates compactness and foreground saliency because the two maps are complementary.The foreground map is itself based on the compactness result.
- S = γ · SCS + (1 −γ) · SF S combines the compactness and foreground maps.The parameter γ balances SCS and SF S.
III. EXPERIMENTAL RESULTS
The method is evaluated on two stereo datasets using precision-recall curves, F-measure, and MAE under fixed parameter settings.
- Evaluation uses NJU-400 and NJU-1985, containing RGB images, depth maps, and pixel-wise ground-truth annotations.These datasets support stereo saliency evaluation with RGB-depth inputs.
- Performance is measured with precision-recall curves, F-measure, and Mean Absolute Error.Precision-recall curves are obtained by thresholding saliency maps from 0 to 255.
- The experiments fix L = 3, T1 = 0.4, T2 = 0.6, and γ = 0.8.
A. Performance Comparison
The proposed method is evaluated against 10 state-of-the-art approaches on NJU-400 and NJU-1985 using quantitative and visual comparisons. It generally achieves stronger performance, while average recall is slightly lower than ACSD on NJU-1985.
- Quantitative comparison: The method is compared with eight 2D and two stereo saliency detection approaches on NJU-400 and NJU-1985.The comparison includes RC, MR, DS, MAP, DCLC, LPS, BSCA, RRWR, SS, and ACSD.
- Quantitative comparison: Precision-recall curves show that the proposed method performs better than the other methods on both datasets.
- Quantitative comparison: The proposed method achieves the best average precision, F-measure, and MAE compared with the other approaches.The authors attribute this result to depth confidence analysis and a two-stage saliency computation mechanism.
- Quantitative comparison: 0.6055 is the proposed method’s F-measure on NJU-1985, compared with 0.5552 for ACSD.
- Quantitative comparison: The proposed method has slightly lower average recall than ACSD on NJU-1985.The paper identifies improving recall while maintaining precision as future work.
- Visual comparison: Visual comparisons include RGB and depth inputs, six competing methods, the proposed saliency map, and ground truth.
B. Parameter Analysis
Factor analysis on NJU-400 examines the effects of depth confidence and the DRSS process using precision-recall curves and quantitative indexes. Both factors improve aspects of final saliency detection.
- Depth confidence: Depth confidence improves performance compared with using no depth confidence measure.This is shown by comparing the black and blue precision-recall curves and the first two columns of quantitative results.
- Evaluation setup: Fig. 6 evaluates different factors on NJU-400 using precision-recall curves and average precision, recall, F-measure, and MAE.
IV. CONCLUSION
The paper presents a stereoscopic saliency model that combines depth confidence analysis with color- and depth-based compactness and multiple-cue foreground saliency. Experiments on two public benchmarks validate the approach’s advantages.
- Conclusion: The model evaluates depth-map reliability with a depth confidence measure before using depth information in saliency detection.The measure is intended to reduce the influence of poor depth maps.
- Conclusion: Color and depth information are integrated into a new compactness model to compute compactness saliency.
- Conclusion: A multiple-cue foreground saliency method uses depth-refined foreground seed selection to improve robustness when compactness saliency makes detection errors.
- Conclusion: The compactness and foreground saliency maps are combined into the final saliency map through a weighted-sum method.
- Conclusion: Experiments on two public benchmarks validate the advantages of the proposed approach.