PaperlayerView exact revision on arXiv

Source-linked AI summary

RGB-D Salient Object Detection: A Survey

Tao Zhou, Deng-Ping Fan, Ming-Ming Cheng, Jianbing Shen, Ling Shao

arXiv:2008.00230v4cs.CV

TL;DR

RGB-D SOD has advanced through depth-enabled models, but the field lacked a comprehensive understanding of its methods, datasets, and challenges. This paper surveys RGB-D and light-field SOD, evaluates representative models, and discusses future directions; deep models outperform traditional ones in the reported evaluation.

  • Problem

    RGB-D SOD methods and challenges lacked an in-depth comprehensive understanding despite promising progress and the availability of useful depth information.

  • Method

    The paper systematically reviews RGB-D models, datasets, light-field SOD, and evaluation practices from multiple perspectives.

  • Results

    Deep learning methods obtain significantly better performance than traditional RGB-D SOD models, while D3Net, JL-DCF, UC-Net, SSF, ICNet, and S2MA obtain the best performance among deep models.

  • Takeaways & Limitations

    The survey organizes the field's models, datasets, evaluations, challenges, and open research directions for future RGB-D SOD work.

Abstract

from arXiv · show

Salient object detection (SOD), which simulates the human visual perception system to locate the most attractive object(s) in a scene, has been widely applied to various computer vision tasks. Now, with the advent of depth sensors, depth maps with affluent spatial information that can be beneficial in boosting the performance of SOD, can easily be captured. Although various RGB-D based SOD models with promising performance have been proposed over the past several years, an in-depth understanding of these models and challenges in this topic remains lacking. In this paper, we provide a comprehensive survey of RGB-D based SOD models from various perspectives, and review related benchmark datasets in detail. Further, considering that the light field can also provide depth maps, we review SOD models and popular benchmark datasets from this domain as well. Moreover, to investigate the SOD ability of existing models, we carry out a comprehensive evaluation, as well as attribute-based evaluation of several representative RGB-D based SOD models. Finally, we discuss several challenges and open directions of RGB-D based SOD for future research. All collected models, benchmark datasets, source code links, datasets constructed for attribute-based evaluation, and codes for evaluation will be made publicly available at https://github.com/taozh2017/RGBDSODsurvey

I. INTRODUCTION

RGB-D SOD extends salient object detection with depth information to address challenging scenes, while this survey organizes models, datasets, evaluations, light-field SOD, and future challenges.

  • I. INTRODUCTION: RGB-D SOD uses depth maps alongside RGB information to improve salient-object detection in challenging scenes.Early methods extracted handcrafted depth-related features and fused RGB images with depth maps.
  • I. INTRODUCTION: It also surveys light-field SOD models and benchmark datasets because light fields provide focal stacks, all-focus images, and depth maps.These additional modalities can provide information useful for salient object detection.
  • I. INTRODUCTION: The survey systematically reviews RGB-D models from traditional/deep, fusion-wise, single-/multi-stream, and attention-aware perspectives.These perspectives address feature extraction, RGB-depth fusion, parameter-accuracy trade-offs, and attention mechanisms.
  • I. INTRODUCTION: The paper reviews nine commonly used RGB-D datasets and evaluates representative models comprehensively and by attributes.The attribute-based evaluation uses newly constructed datasets for representative RGB-D SOD models.
  • I. INTRODUCTION: The survey discusses RGB-D SOD challenges and identifies open directions for future research.Its discussion includes the relationship between SOD and other topics.

C. Organization

The survey organizes RGB-D SOD models by feature extraction, fusion strategy, stream architecture, and attention usage, while also reviewing datasets, light-field SOD, evaluations, and open directions. Fusion methods are grouped into early, late, and multi-scale strategies for combining RGB and depth information.

  • Organization: The survey categorizes RGB-D SOD models as traditional or deep, fusion-wise, single-stream or multi-stream, and attention-aware.These perspectives address feature extraction, modality fusion, computational trade-offs, and attention usage.
  • Traditional/Deep Models: Traditional models use handcrafted depth-related cues, whereas deep models learn high-level representations and complex correlations between RGB images and depth cues.Handcrafted cues include boundaries, shapes, and surface normals; deep models address their limited expression ability.
  • Fusion-wise Models: RGB-D fusion strategies comprise early fusion, late fusion, and multi-scale fusion, each exploring RGB-depth correlations at different processing stages.Figure 3 compares these three strategy families.
  • Fusion-wise Models: Early fusion combines RGB and depth at the input or low-level feature stage, while late fusion combines high-level features or independent saliency maps.The two late-fusion variants are later feature fusion and late result fusion.
  • Fusion-wise Models: Multi-scale fusion models learn cross-modal interactions or combine features from different layers before decoding the final saliency map.Representative designs use interactive conversion, depth-weighted combinations, gated attention, or bilateral attention.

C. Single-stream/Multi-stream Models

RGB-D SOD models use either single-stream or multi-stream architectures. Single-stream designs reduce parameters, whereas multi-stream designs separately process RGB and depth cues to model modality-specific representations.

  • Single-stream Models: Single-stream models fuse RGB images and depth information within the input channels or feature-learning stages to predict saliency.MDSF computes four feature types across three levels and fuses them into the final saliency map.
  • Multi-stream Models: Multi-stream models use independent RGB and depth branches that produce high-level features or saliency maps before middle- or late-stage fusion.Most recent deep RGB-D SOD models use two-stream architectures, while some employ more streams.
  • Single-stream/Multi-stream Trade-offs: Single-stream architectures can save parameters, whereas multiple streams may require more parameters but can model modality correlations across layers.The survey presents this as a balance between computational cost and accuracy.

D. Attention-aware Models

Attention-aware RGB-D SOD methods address unequal regional contributions, cluttered backgrounds, and differing reliability between RGB and depth modalities. The survey also catalogs nine RGB-D datasets with varied sources, scenes, sensors, scales, and challenges.

  • Attention-aware Models: Attention-aware methods respond to the limitation that treating all regions or modalities equally can make RGB-D SOD vulnerable to cluttered backgrounds and unreliable depth.The survey notes that some methods either assign equal status to RGB and depth or overly rely on depth.
  • Attention-aware Models: ASIF-Net weights salient regions through deeply supervised attention while capturing complementary information with interweaved RGB-depth fusion.Its design combines modality complementarity with regional weighting.
  • Attention-aware Models: AttNet uses attention maps to distinguish salient objects from background regions and reduce the influence of low-quality depth cues.This explicitly targets depth-related noise in the prediction process.
  • Attention-aware Models: TANet fuses RGB and depth information from bottom-up and top-down views using channel-wise attention across modalities and feature levels.The channel-wise module is designed to combine complementary information.
  • RGB-D Datasets: The survey reviews nine RGB-D benchmark datasets and provides example RGB images, depth maps, and annotations for each dataset.The datasets vary in scene type, acquisition source, resolution, annotation process, and depth sensor.
  • RGB-D Datasets: SIP contains 929 high-resolution images with multiple salient persons, smartphone-captured depth maps, diverse scenes, challenging factors, and pixel-level ground truths.DUT-RGBD includes indoor and outdoor scenes with transparent objects, complex backgrounds, similar foregrounds and backgrounds, and low-intensity environments.

IV. SALIENCY DETECTION ON LIGHT FIELD

Light-field SOD exploits all-focus images, focal stacks, and rough depth maps to address unreliable or low-quality depth in RGB-D SOD. The survey reviews traditional, deep, and refinement-based light-field models and their benchmark datasets.

  • Motivation: Light-field SOD uses all-focus images, focal stacks, and rough depth maps, providing richer information than depth maps alone.The survey presents light fields as an alternative when inaccurate or low-quality depth maps reduce RGB-D SOD performance.
  • Traditional/Deep Models: Classic light-field SOD models use handcrafted superpixel-level features, including focusness, depth, and objectness cues enabled by refocusing.Some methods also compute background priors from focal slices and combine them with location and contrast cues.
  • Traditional/Deep Models: Deep light-field SOD models include attentive recurrent CNNs, memory-oriented decoders, and feature-integration modules for combining focal-slice or multi-level information.Adversarial examples are used in one model to increase data diversity and robustness.
  • Refinement-based Models: Refinement-based models enforce neighboring constraints or reduce modality homogeneity to improve final light-field saliency predictions.Examples include saliency-dictionary refinement, two-stage refinement, and refinement modules in LFNet.

B. Light Field Data for SOD

The survey reviews five widely used light-field SOD datasets, covering varied scene types, object configurations, imaging conditions, and annotation formats.

  • Five representative datasets are widely used in existing light-field SOD models.They are LFSD, HFUT, DUTLF-FS, DUTLF-MV, and Lytro Illum.
  • LFSD contains 100 light fields spanning indoor and outdoor scenes, with most scenes containing one salient object.Its ground truth was determined from three manual segmentations requiring over 90% overlap.
  • HFUT contains 255 light fields, mostly featuring multiple objects at different locations and scales under complex background clutter.
  • DUTLF-FS provides 1,465 samples with separate training and test sets and challenges including low contrast, disconnected objects, and extreme illumination.The dataset uses 1,000 training samples and 465 test samples at 600 × 400 resolution.
  • DUTLF-MV contains 1,580 samples with multi-view images and corresponding ground-truth annotations, while Lytro Illum contains 640 light fields with per-pixel saliency maps.DUTLF-MV has 1,100 training samples; Lytro Illum includes inconsistent illumination and small objects in similar or cluttered backgrounds.

V. MODEL EVALUATION AND ANALYSIS

The evaluation framework reviews precision-recall, F-measure, MAE, S-measure, and E-measure to assess RGB-D SOD predictions against ground truth across thresholds and image structure.

  • The survey uses five metrics: precision-recall, F-measure, mean absolute error, structural measure, and enhanced-alignment measure.
  • PR: Precision and recall compare a thresholded saliency mask M with ground-truth G, and varying thresholds produces a PR curve.The saliency map is partitioned using thresholds from 0 to 255.
  • F-measure: F-measure combines precision and recall through a weighted harmonic mean, with β^2 = 0.3 emphasizing precision.Fixed thresholds from 0 to 255 yield maximal or average Fβ values.
  • MAE: MAE measures average pixel-wise absolute error between predicted saliency S and ground truth G across all pixels.MAE is normalized to [0,1].
  • S-measure and E-measure: S-measure evaluates structural similarity between regional perception Sr and object perception So, while E-measure combines image-level statistics with local pixel matching.The default S-measure trade-off parameter is α = 0.5, and E-measure uses the enhanced-alignment matrix.

B. Performance Comparison and Analysis

The evaluation compares 24 RGB-D SOD models across multiple datasets and metrics, finding deep methods substantially stronger than traditional methods and identifying six leading deep models.

  • Overall Evaluation: The benchmark evaluates 24 representative models, comprising nine traditional methods and fifteen deep learning-based methods.Mean Sα and MAE are reported across five datasets, with PR and F-measure curves evaluated on eight datasets.
  • Overall Evaluation: Deep learning methods obtain significantly better performance than traditional RGB-D SOD models.The survey attributes this result to the stronger feature-learning ability of deep networks.
  • Overall Evaluation: D3Net, JL-DCF, UC-Net, SSF, ICNet, and S2MA obtain the best performance among the evaluated deep models.
  • Leading Model Designs: D3Net learns modality-specific and fused representations through RgbNet, DepthNet, and RgbdNet, while its depth depurator filters low-quality depth maps.The depurator is intended to prevent unreliable depth from inhibiting RGB-depth fusion.
  • Leading Model Designs: JL-DCF uses joint learning and densely-cooperative fusion to learn robust saliency features and discover complementary RGB-depth features.
  • Leading Model Designs: UC-Net generates multiple saliency predictions by modeling the feature output distribution conditioned on RGB-D images.This models stochastic variation in saliency labeling rather than producing only one deterministic map.

2) Attribute-based Evaluation:

Attribute-based evaluations examine object scale, background clutter, object count, scene type, background objects, and lighting, revealing systematic difficulty patterns and strong recent-model performance.

  • Evaluation Scope: The attribute-based evaluation studies object scale, background clutter, salient-object count, indoor or outdoor scenes, background objects, and lighting conditions.
  • Object Scale: Object scale is defined by salient-area ratios below 0.1, between 0.1 and 0.4, or above 0.4, using a 2,464-image hybrid dataset.Small, medium, and large objects account for 24%, 69.2%, and 6.8% of images, respectively.
  • Object Scale: All comparison methods perform better on small salient objects and worse on large ones; JL-DCF, UC-Net, and S2MA achieve the best scale-related results.
  • Background Clutter: All models perform worse on complex backgrounds than simple ones, with JL-DCF, UC-Net, and SSF achieving the top three results.D3Net, S2MA, A2dele, and ICNet also outperform the other models in this evaluation.
  • Object Count: Single salient objects are easier to detect than multiple salient objects.This comparison uses 1,229 images collected from the NLPR and SIP datasets.
  • Scene Type: Most models struggle more with indoor scenes than outdoor scenes.
  • Background Objects: Performance varies across background-object categories, while JL-DCF, UC-Net, and SSF achieve the top three results.The categories include car, barrier, flower, grass, road, sign, tree, and other.
  • Lighting Conditions: Low-light conditions negatively affect SOD performance, while UC-Net performs best under sunny conditions among the comparison models.

A. Effects of Imperfect Depth

RGB-D SOD performance is constrained by imperfect or missing depth maps, motivating methods that enhance, filter, or estimate depth and handle incomplete modalities.

  • Effects of Low-quality Depth Maps: Depth quality varies across scenarios and directly affects RGB-D SOD performance, making low-quality depth maps a major challenge.Depth maps can provide spatial information for cluttered scenes, but sensor limitations produce variable quality.
  • Effects of Low-quality Depth Maps: Existing approaches enhance depth maps, emphasize unreliable-depth samples, or gate low-quality maps, but often separate enhancement from fusion.These strategies include contrast enhancement, compensation-aware loss, and D3Net’s depth depurator unit.
  • Effects of Low-quality Depth Maps: End-to-end fusion that enhances depth or adaptively downweights poor maps is identified as a need for reducing low-quality-depth effects.The survey characterizes independent gating and two-step enhancement–fusion strategies as potentially suboptimal.
  • Incomplete Depth Maps: Incomplete RGB-D SOD concerns datasets with complete RGB images but missing depth maps after severely noisy or blurred depth samples are discarded.The survey relates this setting to incomplete multi-view or multimodal learning and proposes it as a new direction.
  • Depth Estimation: Depth estimation is proposed as a way to recover higher-quality depth and mitigate low-quality-depth effects in RGB-D SOD.The survey suggests introducing existing depth-estimation approaches into the task.

B. Effective Fusion Strategies

The survey discusses fusion strategies for combining RGB and depth, alternative supervision and modality designs, and dataset and deployment constraints that shape effective RGB-D SOD systems.

  • Fusion Strategies: RGB-D SOD models use early, middle, or late fusion to exploit correlations between RGB images and depth maps.The survey also identifies adversarial and attention-based fusion as promising directions for learning shared or selective representations.
  • Different Supervision Strategies: Weakly supervised, semi-supervised, and self-supervised strategies are proposed to reduce reliance on costly pixel-level saliency annotations.Image-level tags, pseudo pixel-wise annotations, and pretrained self-supervised models are highlighted as possible resources.
  • Dataset Size: RGB-D SOD has nine public datasets, but the largest, NJUD, contains only about 2,000 samples, motivating larger-scale benchmarks.The survey compares this size with RGB-D datasets for generic object detection or action recognition.
  • Complex Background & Task-driven Datasets: Existing datasets often use relatively clean backgrounds, so complex scenes involving occlusion, appearance change, and low illumination are needed for better generalization.The survey identifies task-driven datasets as important for real-world conditions.
  • Model Design for Real-world Scenarios: Complicated deep networks limit direct deployment on real-world platforms, motivating model compression and compact shared-network designs.JL-DCF is cited as using a shared network to reduce parameters and support real-world applications.
  • Related Multimodal Directions: RGB-T SOD supplements RGB with thermal infrared data that is insensitive to illumination, targeting varying light, reflective light, or shadows.The survey briefly discusses RGB-T SOD as an extension related to multimodal detection.
Loading 2008.00230v4…