PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks

Deng-Ping Fan, Zheng Lin, Jia-Xing Zhao, Yun Liu, Zhao Zhang, Qibin Hou, Menglong Zhu, Ming-Ming Cheng

arXiv:1907.06781v2cs.CV

TL;DR

RGB-D salient object detection evaluations rely heavily on pixel-level metrics, making thorough and reliable conclusions difficult. The paper introduces the SIP dataset, D3Net, and a large benchmark; D3Net outperforms prior SOD approaches and is practical for RGB-D applications.

  • Problem

    Existing RGB-D salient object detection evaluations rely on pixel-level F-measure, MAE, and precision-recall metrics, making thorough and reliable conclusions difficult.

  • Method

    The paper introduces the human-oriented SIP dataset, proposes D3Net with depth filtering and cross-modal feature learning, and conducts a large-scale RGB-D SOD benchmark.

  • Results

    D3Net outperforms prior SOD approaches without post-processing, while the study benchmarks about 97K images across 18 models.

  • Takeaways & Limitations

    The benchmark and publicly released resources support further RGB-D SOD research, while D3Net offers a practical solution for real-world applications.

  • Takeaways & Limitations

    The SIP dataset is relatively small compared with most RGB salient object detection datasets and is planned for expansion.

Abstract

from arXiv · show

The use of RGB-D information for salient object detection has been extensively explored in recent years. However, relatively few efforts have been put towards modeling salient object detection in real-world human activity scenes with RGBD. In this work, we fill the gap by making the following contributions to RGB-D salient object detection. (1) We carefully collect a new SIP (salient person) dataset, which consists of ~1K high-resolution images that cover diverse real-world scenes from various viewpoints, poses, occlusions, illuminations, and backgrounds. (2) We conduct a large-scale (and, so far, the most comprehensive) benchmark comparing contemporary methods, which has long been missing in the field and can serve as a baseline for future research. We systematically summarize 32 popular models and evaluate 18 parts of 32 models on seven datasets containing a total of about 97K images. (3) We propose a simple general architecture, called Deep Depth-Depurator Network (D3Net). It consists of a depth depurator unit (DDU) and a three-stream feature learning module (FLM), which performs low-quality depth map filtering and cross-modal feature learning respectively. These components form a nested structure and are elaborately designed to be learned jointly. D3Net exceeds the performance of any prior contenders across all five metrics under consideration, thus serving as a strong model to advance research in this field. We also demonstrate that D3Net can be used to efficiently extract salient object masks from real scenes, enabling effective background changing application with a speed of 65fps on a single GPU. All the saliency maps, our new SIP dataset, the D3Net model, and the evaluation tools are publicly available at https://github.com/DengPingFan/D3NetBenchmark.

I. INTRODUCTION

RGB-D salient object detection is motivated by smartphone depth sensing but remains limited by unrealistic human-scene data, noisy depth maps, narrow evaluation, and inconsistent metrics.

  • Depth information is widely available in modern smartphones, but most salient object detection methods rely only on RGB images.
  • Existing RGB-D datasets generally use Kinect, light-field, or optical-flow-derived depth rather than smartphone captures, leaving realistic human-centered scenes underrepresented.
  • Low-quality depth maps can introduce noise and wrong detections, yet existing models do not explicitly discard unreliable depth information.
  • Benchmarking only 1–4 datasets makes model generalizability difficult to assess.
  • Pixel-level metrics such as F-measure, MAE, and PR curves alone make thorough quantitative conclusions difficult.
  • Unreported F-measure statistics and differing threshold strategies can produce unfair and inconsistent comparisons.

A. Contribution

The paper contributes a smartphone-oriented salient-person dataset, a broad benchmark, and D3Net, a depth-filtering RGB-D model supported by evaluation tools and analyses.

  • The SIP dataset contains 929 high-resolution images captured with a real smartphone and annotated for challenging human-centered scenes.It includes diverse scenes and situations such as occlusion and appearance change, with pixel-level ground truths.
  • SIP additionally provides RGB and grayscale images from a binocular camera, supporting directions including stereo matching and depth estimation.
  • The benchmark compares 32 RGB-D salient object detection models and evaluates 18 state-of-the-art algorithms across SIP and six existing datasets.The study also provides an online evaluation platform with a preserved test set.
  • D3Net combines a three-stream feature learning module with a depth depurator unit that automatically filters low-quality depth maps.The framework is designed to learn cross-modality features from RGB images and depth maps.
  • The work provides systematic benchmark tools and analyses intended to support comprehensive assessment and future RGB-D research.The paper describes quantitative and qualitative experiments and runtime analysis.

II. RELATED WORKS

Prior RGB-D salient object detection work spans specialized datasets and hand-crafted or CNN-based models, while this paper targets smartphone-realistic human scenes and noisy depth.

  • Earlier RGB-D datasets differ in purpose and scale, including stereoscopic, light-field, generic-object, and indoor collections.
  • Existing datasets generally lack depth maps captured by real smartphones, limiting their reflection of environmental conditions such as lighting and object distance.
  • SIP contains 929 outdoor images covering challenging cases including dark backgrounds, occlusion, appearance change, and out-of-view persons.
  • SIP uses smartphone dual-camera RGB, grayscale, and estimated-depth captures while emphasizing salient persons in real-world scenes.
  • Its dataset analysis includes center bias and object-size distribution, aspects not carefully investigated in prior RGB-D studies.
  • Traditional methods use hand-crafted contrast, shape, spatial, geometric, background, graph, or probabilistic features for RGB-D saliency.
  • CNN-based methods improve expressive capacity by learning salient objects from RGB-D data, but D3Net specifically excludes low-quality depth maps during complementary feature learning.

III. PROPOSED DATASET

SIP is a human-activity-oriented RGB-D salient person dataset built from smartphone imagery, emphasizing realistic scenes, challenging conditions, and multiple salient objects. Its statistics characterize object placement, size, and scene diversity.

  • Dataset Overview: SIP contains 929 RGBD images from eight background scenes, portraying multiple actors with varied clothing and challenging cases including appearance change, occlusion, and shape complexity.The images were collected with a Huawei Mate 10, whose depth maps were automatically estimated.
  • Dataset Construction: 929 high-resolution images were selected from 5,269 captures after manual screening and voting-based annotation procedures focused on salient people.Six viewers drew bounding boxes around the most attention-grabbing person, with low-consistency images discarded.
  • Dataset Statistics: Unlike most compared datasets, SIP and two small-scale datasets show limited center bias, whereas most others place object centers close to the image center.The comparison uses normalized distances of object centers and farthest object-margin points from the image center.
  • Dataset Statistics: SIP object sizes range from 0.48% to 66.85% of image pixels, averaging 20.43%, while background objects include cars, trees, and grass.Object size is defined as the ratio of salient-object pixels to total image pixels.
  • Challenging Conditions: SIP includes dark and clear object-boundary conditions, with low-light depth maps introducing additional challenges for salient-object detection.The dataset also covers diverse outdoor scenarios with dark backgrounds, occlusion, appearance change, and out-of-view cases.
  • Salient Object Instances: SIP was designed to include up to five salient objects per image, addressing the limited object counts of existing datasets.The design follows findings that humans can enumerate at least five objects without counting.

IV. PROPOSED MODEL

D3Net combines three-stream cross-modal feature learning with a depth depurator unit that filters low-quality depth information. Its nested architecture uses pyramid features and jointly learns from RGB and depth inputs.

  • Architecture: D3Net contains a three-stream feature learning module and a depth depurator unit that extracts cross-modal features and filters low-quality depth maps.The DDU acts as a gate, allowing RGB-only flow when it filters out a depth map.
  • Feature Learning Module: The feature learning module uses RgbNet, RgbdNet, and DepthNet with the same structure but different input channels to produce three saliency maps.Each sub-network receives a re-scaled 224×224 image and predicts one of Srgb, Srgbd, or Sdepth.
  • Feature Learning Module: The three subnetworks use a modified Feature Pyramid Network with bottom-up and top-down pathways plus lateral connections for multi-level feature extraction.The design adds a sixth layer to VGG-16 and merges information by concatenation rather than addition.
  • Feature Learning Module: The pyramid construction upsamples coarser features by 2×, reduces finer-map channels with 1×1 convolution, and concatenates the maps.For RgbdNet, the input is a four-dimensional feature tensor and the resulting pyramid contains features at multiple spatial resolutions.

B. Depth Depurator Unit (DDU)

DDU uses a gate connection to select an optimal saliency prediction by rejecting low-quality depth maps. It compares predictions from RGB-depth and depth-only streams, motivated by their differing similarity under depth quality.

  • DDU motivation: Low-quality depth maps can introduce more noise than useful cues, so the gate connection classifies and excludes poor depth inputs.The gate is used in the test phase rather than training.
  • DDU motivation: High-quality depth maps typically yield closed object boundaries and clear double peaks, whereas low-quality maps contain cluttered elements.These depth-distribution characteristics motivate automatic quality rejection.
  • DDU operation: The DDU receives three predicted maps from RGB, RGB-depth, and depth streams and decides which prediction P is optimal.Inputs are resized to reduce computational complexity before prediction selection.
  • DDU operation: The comparison unit Fcu assesses similarity between Sdepth and Srgbd and acts as a gate selecting the RgbNet or RgbdNet prediction.The DDU is implemented as a post-processing scheme during testing.
  • DDU limitation: Fcu is treated as a fixed weight, while an adaptive-weight formulation is identified as future work.The complementary weight is defined as ¯Fcu = 1−Fcu.

C. Implementation Details

D3Net uses a mean-absolute-error-based depth-quality comparison with a fixed threshold and is trained with cross entropy under specified dataset and optimization settings. The benchmark covers approximately 97K image-model evaluations.

  • DDU: The DDU uses mean absolute error to compare Srgbd and Sdepth, with threshold t = 0.15 achieving the best tested performance.The tested threshold values were 0.01, 0.02, 0.05, 0.10, 0.15, and 0.20.
  • Loss function: Cross entropy loss trains saliency maps estimated from RGB, RGB-depth, or depth inputs against ground-truth maps.The estimated and ground-truth maps are represented at 224×224 resolution.
  • Training settings: 2,185 image pairs from NJU2K and NLPR form the training data, using 1,485 and 700 pairs respectively.The training follows the settings described in prior work for fair comparison.
  • Training settings: Training uses Adam with an initial learning rate of 1e-4, batch size 8, and 30 epochs on a GTX TITAN X GPU.The implementation uses Python and PyTorch.
  • Data augmentation: Horizontal image flipping augments training samples to reduce overfitting risk caused by limited dataset scale.
  • Benchmark scale: 97K images are benchmarked across 5,398 images and 18 models, making this the largest and most comprehensive RGB-D SOD benchmark to date.

A. Experimental Settings

The evaluation compares 18 traditional and CNN-based models across seven RGB-D datasets using five complementary metrics and standardized runtime measurements. It includes both saliency quality and computational efficiency.

  • Models and datasets: The benchmark evaluates 18 state-of-the-art models, comprising 10 traditional methods and 8 CNN-based methods.
  • Models and datasets: Seven datasets provide test data, including STERE, LFSD, DES, NLPR, NJU2K, SSD, and SIP.
  • Runtime: D3Net requires about 0.015s per 224×224 image on the reported CPU-GPU platform without post-processing such as CRF.Runtime comparisons use the same Intel Xeon E5-2676v3 and GTX TITAN X platform.
  • Evaluation metrics: MAE measures the average pixelwise difference between a real-valued saliency map and binary ground truth, normalized to [0,1].It gives larger objects larger errors and does not indicate where errors occur.
  • Evaluation metrics: PR curves vary a fixed saliency threshold from 0 to 255 and plot the resulting precision-recall pairs.
  • Evaluation metrics: F-measure is evaluated as the maximum region-based similarity over fixed thresholds from 0 to 255.
  • Evaluation metrics: S-measure combines region-aware and object-aware structural similarity, while E-measure combines image-level statistics with local pixel matching.

C. Metric Statistics

The benchmark summarizes metric statistics across datasets and finds D3Net, CPFP, and TANet among the leading deep models. Deep methods generally outperform traditional algorithms, with dataset-specific exceptions.

  • Metric statistics: For each metric, mean statistics average image-level scores over all images in a dataset.The resulting means across datasets are summarized in Table IV.
  • Traditional models: SE, MDSF, and DCMC are identified as the top three traditional algorithms by overall Table IV performance.SE and DCMC use superpixel-based RGB region contrast, while MDSF formulates SOD as pixelwise binary labeling with SVM.
  • Deep models: D3Net, CPFP, and TANet are the top three deep models among the leading methods.The authors associate this result with the feature representation ability of deep learning for RGB-D SOD.
  • Traditional versus deep models: Most deep models perform better than traditional algorithms overall, although MDSF outperforms DF and AFNet on NLPR.

E. Comparison with SOTAs

D3Net is compared with leading RGB-D salient object detection methods using benchmark evaluations and challenging visual scenes. It achieves higher performance than prior contenders and handles damaged or unreliable depth maps through adaptive modality selection.

  • D3Net outperforms the best published result, CPFP, by 1.0%–5.8% on six datasets and improves 1.4% on SIP.The comparison evaluates D3Net against 17 state-of-the-art models.
  • D3Net remains effective when depth structures are partially or severely damaged, locally connected to backgrounds, or paired with multiple isolated salient objects.It filters low-quality depth maps and selects complementary RGB-depth cues to preserve salient-object details.
  • Visual comparisons cover 18 methods across NJU2K, STERE, and NLPR using precision-recall curves and F-measures under fixed thresholds.
  • On SIP, precision-recall curves and F-measures are compared under different thresholds.
  • RGB-D salient object detection supports smartphone functions such as object extraction, bokeh effects, and mobile user recognition, as well as suspicious-object discovery.

B. Background Changing Application

The background-changing application uses RGB-D saliency to preserve salient regions while adapting images for book-cover layouts. The paper presents a prototype rather than a complete software system.

  • The prototype addresses time-consuming, technically demanding Photoshop-based book-cover design with an easier background-changing workflow.
  • Users upload a candidate design image, after which content-based features including an RGB-D saliency map support background-changing operations.
  • The input image is resized to match the target style size while preserving the salient region according to D3Net inference.
  • The paper does not focus on designing a complete software system and suggests future systems could associate background images with specified topics.
  • D3Net is evaluated through comparisons, ablations, and benchmarking that examine RGB-depth fusion, depth filtering, and performance across datasets.
  • The SIP dataset comparison emphasizes smooth, fine object- and instance-level boundaries and explicitly considers occlusions.

B. Limitations

The paper identifies limitations involving SIP’s scale, D3Net’s memory demands, and the DDU’s gap from its optimal upper bound. It outlines dataset expansion and lightweight-model alternatives as future directions.

  • SIP is relatively small compared with most RGB salient object datasets, although it targets smartphone-based real human activity applications.
  • The authors plan to enlarge SIP with more challenging situations and varied foreground persons.
  • D3Net’s three subnetworks may increase memory usage on lightweight devices.Suggested alternatives include MobileNet V2, dimension reduction, and ESPNet V2.
  • The DDU does not reach the best upper bound on the current training subset, leaving room for improved depth-depurator designs.
  • The benchmark includes 32 summarized state-of-the-art models and 18 evaluated traditional or deep models across approximately 97K images.
  • The study expects its benchmark and dataset to support future work on real-world multi-person and complex-interaction scenarios using mobile devices.
Loading 1907.06781v2…