PaperlayerView exact revision on arXiv

Source-linked AI summary

Review of Visual Saliency Detection with Comprehensive Information

Runmin Cong, Jianjun Lei, Huazhu Fu, Ming-Ming Cheng, Weisi Lin, Qingming Huang

arXiv:1803.03391v2cs.CV

TL;DR

Visual saliency detection must extend beyond RGB appearance to use depth, inter-image correspondence, and temporal information for RGBD, co-saliency, and video tasks. This survey synthesizes methods, evaluation practices, challenges, and future directions across these settings. It reports that appropriate comprehensive information generally improves performance, while depth quality remains a limiting factor.

  • Problem

    Existing saliency tasks require incorporating depth, inter-image correspondence, or temporal and motion information to identify salient objects in richer visual settings.

  • Method

    The paper surveys RGBD, co-saliency, and video saliency detection methods, evaluation datasets, quantitative measures, experimental comparisons, and open problems.

  • Results

    The survey reports improved performance when comprehensive information is introduced appropriately; depth-measure methods often perform better, and combining cues benefits RGBD co-saliency and deep video methods.

  • Takeaways & Limitations

    Saliency models can be extended by integrating depth, correspondence, motion, and spatiotemporal constraints, with their effectiveness depending on how these cues are exploited.

  • Takeaways & Limitations

    Poor depth maps can degrade saliency measurement, making depth quality a practical limitation for RGBD saliency detection.

Abstract

from arXiv · show

Visual saliency detection model simulates the human visual system to perceive the scene, and has been widely used in many vision tasks. With the acquisition technology development, more comprehensive information, such as depth cue, inter-image correspondence, or temporal relationship, is available to extend image saliency detection to RGBD saliency detection, co-saliency detection, or video saliency detection. RGBD saliency detection model focuses on extracting the salient regions from RGBD images by combining the depth information. Co-saliency detection model introduces the inter-image correspondence constraint to discover the common salient object in an image group. The goal of video saliency detection model is to locate the motion-related salient object in video sequences, which considers the motion cue and spatiotemporal constraint jointly. In this paper, we review different types of saliency detection algorithms, summarize the important issues of the existing methods, and discuss the existent problems and future works. Moreover, the evaluation datasets and quantitative measurements are briefly introduced, and the experimental analysis and discission are conducted to provide a holistic overview of different saliency detection methods.

I. INTRODUCTION

Visual saliency detection models identify visually important regions for human-inspired perception and computer-vision applications. Additional depth, inter-image, and temporal information extends image saliency detection to RGBD, co-saliency, and video settings.

  • Visual saliency models allocate attention to attractive or interesting regions and support tasks including segmentation, retrieval, retargeting, compression, and video summarization.
  • Image saliency detection methods are classified into bottom-up stimulus-driven models using low-level features and top-down models.
  • RGBD saliency detection combines color and depth information, enabled by depth acquisition technologies such as structured light, TOF, and binocular imaging.
  • Co-saliency detection finds common salient regions across related images using inter-image correspondence, while requiring saliency, repetition, and appearance similarity.
  • Video saliency detection jointly considers spatial information, temporal constraints, and motion cues to continuously locate motion-related salient objects.
  • Image saliency detection forms the basis for RGBD, co-saliency, and video models, which add depth, inter-image, or temporal information and motion cues.
  • The survey reviews these model families, evaluation datasets, quantitative measurements, experimental comparisons, existing problems, and future work.

II. RGBD SALIENCY DETECTION

RGBD saliency detection supplements color-based analysis with depth information, either as an explicit feature or through depth-derived measures. Existing methods use handcrafted and learned fusion strategies, but depth quality and effective depth exploitation remain important concerns.

  • RGBD saliency detection uses depth either as an additional feature or through depth measures that exploit shape, contour, and other implicit attributes.
  • Depth-measure methods can exploit depth shape to highlight salient objects, depth boundaries to sharpen object contours, and depth consistency to improve map smoothness.
  • Depth-feature methods directly embed depth into the feature pool, combining color, luminance, texture, and depth or fusing multi-level saliency measures.
  • CNN-based RGBD methods learn interactions between low-level cues and saliency, then can refine initial maps with Laplacian propagation.
  • Effective depth exploitation remains unresolved, motivating attempts to combine explicit and implicit depth information into a more comprehensive representation.

B. Depth Measure Based RGBD Saliency Detection

Depth-measure methods exploit implicit shape and contour attributes in depth maps to refine RGBD saliency, while depth quality remains a practical constraint. The section also situates co-saliency methods around inter-image correspondence and learning-based representations.

  • Depth Measure Based RGBD Saliency Detection: Depth-measure methods exploit implicit depth attributes, such as shape and contour, to refine saliency beyond simply adding depth as a feature.Examples include anisotropic center-surround difference, local background enclosure, and depth contrast measures.
  • Depth Measure Based RGBD Saliency Detection: Depth confidence uses depth-map statistics to control depth-information introduction and combine depth-aware compactness with depth-guided foreground saliency.The measure uses mean depth, coefficient of variation, and depth frequency entropy to distinguish depth-map quality.
  • RGB Co-saliency Detection: Co-saliency detection uses inter-image correspondence to identify common salient regions and can be organized by matching, clustering, rank analysis, propagation, or learning strategies.These strategies model relationships among image units using different correspondence-capturing mechanisms.
  • RGB Co-saliency Detection: Learning-based co-saliency methods use high-level semantic representations, including convolutional features, group-wise interactions, multi-instance learning, and metric learning.The reviewed approaches aim to represent both individual images and relationships within image groups.

B. RGBD Co-saliency Detection

RGBD co-saliency detection jointly combines depth constraints with inter-image correspondence, but remains limited by the small number of available methods and datasets. Reviewed approaches use depth features, depth measures, and iterative refinement.

  • RGBD Co-saliency Detection: RGBD co-saliency detection jointly combines depth constraints with inter-image correspondence to identify common salient regions.The task differs from RGB co-saliency by requiring both depth and cross-image information.
  • RGBD Co-saliency Detection: Only a few RGBD co-saliency methods have been proposed, with RGBD Coseg183 and RGBD Cosal150 identified as commonly used datasets.The passage attributes the limited method count to constraints in available data sources.
  • RGBD Co-saliency Detection: Existing methods use depth features such as average depth, depth range, and depth-map HOG to represent regional depth attributes.These features can support feature bagging, regional clustering, and correspondence modeling.
  • RGBD Co-saliency Detection: An iterative framework combines addition, deletion, and refinement cycles to introduce depth information, capture inter-image correspondence, and improve saliency maps.Its Depth Shape Prior descriptor exploits depth shape and converts RGB saliency maps into RGBD saliency maps.

C. Discussions

Video saliency research focuses mainly on low-level cues, while co-saliency depends critically on accurate inter-image correspondence. The discussion highlights computational and modeling challenges alongside the difficulty of exploiting depth effectively.

  • Discussions: Accurate inter-image correspondence can eliminate non-common saliency interference, whereas inaccurate correspondence can degenerate co-saliency performance.Matching- and propagation-based methods are described as relatively accurate but time-consuming.
  • Discussions: Learning-based co-saliency still faces challenges in modeling relationships among multiple images, and simple stack strategies may not improve performance.The discussion identifies multi-image inter-image modeling as an open issue.
  • Discussions: Exploiting depth attributes to improve identification of co-salient objects remains an area for further investigation.This limitation is stated alongside concerns about correspondence accuracy and computational cost.
  • Video Saliency Discussion: Most video saliency methods exploit low-level color, motion, and prior cues, while only a few learn high-level features through networks.The reviewed video setting also involves repeated, motion-related, and distinctive targets.

A. Low-level Cue Based Video Saliency Detection

Low-level video saliency methods extract spatial and temporal cues either separately and fuse them or directly model spatiotemporal features. The review characterizes fusion as intuitive and widely used, while direct pipelines provide a branch-free alternative.

  • Fusion Model: Fusion models separately extract spatial and temporal saliency and combine them into a final spatiotemporal saliency map.Spatial cues describe intra-frame information, while temporal cues describe inter-frame relationships.
  • Direct-Pipeline Model: Direct-pipeline models progressively extract spatiotemporal features to generate final saliency without separate branches.Examples use temporal slices, low-rank and sparse decomposition, optical flow, gradient flow fields, or energy optimization.
  • Fusion Model: Spatial saliency methods reuse image-saliency techniques such as sparse reconstruction, low-rank analysis, contrast priors, and background priors.These methods model intra-frame appearance and structural cues.
  • Fusion Model: Temporal saliency methods use motion information including neighboring-frame reconstruction, bitstream motion vectors, motion histograms, and flow-based consistency.These cues represent moving objects and inter-frame changes.
  • Discussions: Fusion is described as more intuitive than direct pipelines, and existing image-saliency methods can be directly reused for spatial saliency.The review states that most methods focus on the fusion category.

B. Learning Based Video Saliency Detection

Learning-based video saliency detection includes supervised and unsupervised approaches that model spatiotemporal information. The survey emphasizes motion cues, temporal consistency, and efficiency as central design considerations.

  • Video saliency methods are broadly divided into supervised and unsupervised learning approaches.
  • Supervised methods learn spatiotemporal features from large collections of labelled video sequences.One described model combines local region-based CNN features with global block-based CNN features and temporal-segment embeddings.
  • DSRFCN3D uses an end-to-end 3D fully convolutional encoder-decoder network to extract deep features from video blocks and learn contextual information.Its loss jointly combines saliency prediction and 3D deconvolution losses.
  • An unsupervised approach combines pixel-, superpixel-, and object-level spatiotemporal-neighbor cues before using stacked autoencoders to obtain an initial saliency map.Post-processing is then applied to refine the result.
  • Motion cues help suppress backgrounds and static salient objects, while appearance consistency across frames is improved using techniques such as energy-function optimization.Deep networks may learn motion directly from consecutive frames to avoid the time cost and possible inaccuracy of optical-flow calculation.

V. EVALUATION AND DISCUSSION

The survey evaluates saliency maps using threshold-based, curve-based, and direct-error measures. PR/F-measure, ROC/AUC, and MAE quantify complementary aspects of agreement with ground truth.

  • Evaluation metrics include PR curves, F-measure, ROC curves, AUC scores, and MAE.These measures quantitatively compare predicted saliency maps with ground-truth annotations.
  • Precision-Recall curve and F-measure: PR curves are generated by thresholding saliency maps from 0 to 255 and plotting precision against recall; curves nearer the upper left indicate better performance.
  • Precision-Recall curve and F-measure: F-measure combines precision and recall through a weighted harmonic mean, with β2 generally set to 0.3 to emphasize precision.
  • ROC curve and AUC score: ROC curves describe false positive rate versus true positive rate; curves nearer the upper right and larger AUC scores indicate better performance.
  • Mean Absolute Error score: MAE directly measures the difference between the continuous saliency map and ground truth, with smaller values indicating greater similarity.

B. Datasets

The survey reviews datasets for RGB, RGBD, co-saliency, and video saliency detection. RGBD datasets remain relatively limited, while video and co-saliency benchmarks provide varied annotated sequences and image groups.

  • Image saliency detection: Image-saliency benchmarks include large pixel-level datasets such as DUT-OMRON, MSRA10K, HKU-IS, and XPIE.
  • RGBD saliency detection: RGBD saliency evaluation relies mainly on NLPR and NJUD, containing 1,000 and 2,000 RGBD images, respectively.NLPR images have 640 × 640 resolution, while NJUD images have 600 × 600 resolution.
  • Co-saliency detection: Co-saliency evaluation commonly uses five RGB datasets and two RGBD datasets, including MSRC, iCoseg, RGBD Coseg183, and RGBD Cosal150.
  • Co-saliency detection: RGBD Coseg183 contains 183 images in 16 groups, whereas RGBD Cosal150 contains 150 images in 21 groups.
  • Video saliency detection: Video-saliency datasets include ViSal, MCL, UVSD, VOS, SegTrackV1, SegTrackV2, and DAVIS.DAVIS contains 50 fully annotated video sequences, while UVSD contains 18 unconstrained videos with complicated motion and cluttered scenes.

C. Comparison and Analysis

The comparisons show how depth, inter-image correspondence, and temporal or motion cues affect saliency detection across RGBD, co-saliency, and video settings. Deep learning methods often perform strongly, while data availability, depth quality, correspondence, and motion modeling remain important constraints.

  • Image Saliency Detection vs RGBD Saliency Detection: Depth cues improve salient-object consistency and suppress background false positives in RGBD saliency detection.DCMC suppresses background trees more effectively than several RGB methods in the NJUD examples.
  • Image Saliency Detection vs RGBD Saliency Detection: Deep learning RGB methods achieve the top two PR and ROC performances on NJUD and NLPR, while DCMC reaches third on the ROC curve.The reported measures include F-measure, AUC, and MAE, with quantitative trends consistent with visualization results.
  • Image Saliency Detection vs RGBD Saliency Detection: An 18% maximum F-measure gain is reported on NJUD, while RGBD performance varies with depth modeling, depth quality, and labelled-data availability.ACSD captures depth attributes, DCMC adds depth confidence for robustness, and DF is limited by scarce labelled RGBD data.
  • Image Saliency Detection vs Co-saliency Detection: Inter-image correspondence helps co-saliency methods suppress shared backgrounds, and adding depth produces gains exceeding 30% in RGBD co-saliency.MCLP more consistently highlights the common computer while eliminating backgrounds and non-common objects, though similar-colored flashlights remain mistakenly reserved.
  • Image Saliency Detection / Co-saliency Detection vs Video Saliency Detection: Video saliency methods outperform unsupervised image and co-saliency methods on UVSD and DAVIS by using temporal and motion information.Video methods highlight moving objects more homogeneously and suppress backgrounds, whereas image and co-saliency methods lack sufficient temporal or motion constraints.
  • Image Saliency Detection / Co-saliency Detection vs Video Saliency Detection: On UVSD, F-measure rises from 0.3059 to 0.6016, a 96.7% gain; on DAVIS, the maximum reported F-measure gain reaches 52.5%.VFCN can partially suppress static objects through motion cues and inter-frame continuity, while DSS may retain a static bench.

D. Applications

The survey illustrates two applications that extend saliency detection beyond conventional images: remote-sensing ROI extraction and video primary-object detection. Video detection additionally requires temporal reasoning because primary objects may vary in saliency and scenes involve motion.

  • Applications: Saliency detection supports ROI extraction in remote-sensing images and primary-object detection in video.The former resembles image saliency detection, while the latter is analogous to video saliency detection.
  • ROI Extraction in Remote Sensing Image: Remote-sensing ROI extraction produces saliency masks for high-resolution images captured mainly from a high-angle viewpoint.Such images contain many small targets.
  • Primary Object Detection in Video: Video primary-object segmentation obtains pixel-level results, whereas primary-object discovery locates the object with a bounding box.The detection flowchart includes saliency-model construction and primary-object detection.
  • Primary Object Detection in Video: Primary objects may not be the most salient in every frame, so inter-frame correspondence helps highlight them and suppress distractors.Video detection must remain consistent across changing scenes, unlike salient-object detection on an individual fixed scene.
  • Primary Object Detection in Video: Camera and object motion can invalidate visual priors commonly used for image saliency, including the background prior.Motion cues and global appearance should jointly constrain initial result generation.

E. Challenges and Problems

The survey identifies unresolved challenges in representing depth, modeling inter-image correspondence, and combining motion with temporal constraints. These issues concern accurate information extraction, computational costs, and robust saliency across complex or changing data.

  • Overall Challenges: Saliency detection continues to contain unresolved issues requiring further investigation.The survey frames these issues as future research problems across saliency-detection settings.
  • RGBD Saliency Detection: RGBD saliency detection must capture accurate and effective depth representations rather than treating depth only as an additional color-like feature.Depth shape, boundaries, and consistency can provide complementary cues for salient-object highlighting and sharper maps.
  • RGBD Saliency Detection: Depth-based methods can exploit shape and boundary attributes to suppress background, refine object boundaries, and improve saliency-map consistency.The survey states that depth-measure methods generally achieve better performance, but effective exploitation lacks consensus.
  • Co-saliency Detection: Co-saliency detection must discover inter-image correspondence to constrain the common salient object across multiple images.Existing formulations include clustering, matching, propagation, and learning, but may be noise-sensitive or time-consuming.
  • Video Saliency Detection: Video saliency detection must jointly combine motion cues, inter-frame correspondence, and spatiotemporal consistency.These constraints address salient-object discovery in clustered, complex scenes and capture common attributes across the video.

VI. CONCLUSION AND FUTURE WORK

The paper surveys saliency detection with depth, inter-image, and temporal information, combining algorithmic review with experimental comparison. It reports improved performance when comprehensive information is introduced appropriately and identifies learning-based directions for future work.

  • Conclusion: The survey covers RGBD, co-saliency, and video saliency detection through depth cues, inter-image correspondence, and temporal constraints.It reviews recent progress, analyzes algorithm types, and conducts experimental comparisons.
  • Conclusion: Introducing comprehensive information appropriately improves saliency-detection performance across the reviewed settings.The reported evidence includes depth, correspondence, and temporal information as extensions of image saliency detection.
  • Conclusion: Depth-measure methods often perform better because they capture effective depth attributes from the depth map.This finding concerns RGBD saliency detection specifically.
  • Conclusion: Combining inter-image correspondence with depth cues yields better performance in RGBD co-saliency detection models.The conclusion identifies this combination as a reported performance improvement.
  • Conclusion: Deep video-saliency methods use symmetrical structures and multiple-frame inputs to learn high-level spatiotemporal features and improve efficiency.Examples include convolution-deconvolution and encoder-decoder structures.
  • Future Work: Future work includes small-sample training, weakly supervised learning, and cross-domain learning for saliency detection.The survey links these directions to limitations in labeled training data and the goal of high-precision detection.
Loading 1803.03391v2…