PaperlayerView exact revision on arXiv

Source-linked AI summary

Promoting Saliency From Depth: Deep Unsupervised RGB-D Saliency Detection

Wei Ji, Jingjing Li, Qi Bi, Chuan Guo, Jie Liu, Li Cheng

arXiv:2205.07179v1cs.CV

TL;DR

RGB-D saliency detection has often required pixel-level training annotations, while unsupervised RGB-D methods have relied on handcrafted features. This paper refines pseudo-labels using disentangled depth and attentive training; it reports strong benchmark performance and real-time speed. The approach can also improve existing supervised RGB-D models when augmented with DSU.

  • Problem

    RGB-D SOD methods often require labor-intensive pixel-level annotations, while unsupervised alternatives primarily rely on handcrafted features.

    Source: §1 · pp. 1–2

  • Method

    The approach iteratively refines pseudo-labels using depth-disentangled saliency cues and re-weights training samples to emphasize more reliable labels.

    Source: §1 · p. 2

  • Results

    The approach outperforms state-of-the-art unsupervised methods on four public benchmarks and runs at 35 FPS.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Adding DSU to existing RGB-D SOD models notably improves their empirical results in fully supervised settings.

    Source: §1 · p. 2

    Without accurate pixel-level annotations, the model still fails to comprehensively detect fine-grained details such as salient-object edges.

    Source: §A.4 · p. 17

Abstract

from arXiv · show

Growing interests in RGB-D salient object detection (RGB-D SOD) have been witnessed in recent years, owing partly to the popularity of depth sensors and the rapid progress of deep learning techniques. Unfortunately, existing RGB-D SOD methods typically demand large quantity of training images being thoroughly annotated at pixel-level. The laborious and time-consuming manual annotation has become a real bottleneck in various practical scenarios. On the other hand, current unsupervised RGB-D SOD methods still heavily rely on handcrafted feature representations. This inspires us to propose in this paper a deep unsupervised RGB-D saliency detection approach, which requires no manual pixel-level annotation during training. It is realized by two key ingredients in our training pipeline. First, a depth-disentangled saliency update (DSU) framework is designed to automatically produce pseudo-labels with iterative follow-up refinements, which provides more trustworthy supervision signals for training the saliency network. Second, an attentive training strategy is introduced to tackle the issue of noisy pseudo-labels, by properly re-weighting to highlight the more reliable pseudo-labels. Extensive experiments demonstrate the superior efficiency and effectiveness of our approach in tackling the challenging unsupervised RGB-D SOD scenarios. Moreover, our approach can also be adapted to work in fully-supervised situation. Empirical studies show the incorporation of our approach gives rise to notably performance improvement in existing supervised RGB-D SOD models.

1 INTRODUCTION

The introduction frames deep unsupervised RGB-D saliency detection as an alternative to pixel-annotated training and handcrafted unsupervised features. It proposes depth-informed pseudo-label refinement and attentive training, reporting benchmark gains and efficient RGB-only inference.

1 INTRODUCTION

  • Existing RGB-D SOD methods often require extensive pixel-level annotations, while unsupervised alternatives rely primarily on handcrafted features.The task identifies and segments distinctive objects using an RGB image and its depth map.Source: §1 · p. 1
  • Raw depth maps can fail to separate salient objects from their surroundings, while unreliable pseudo-labels can introduce false positives and damage predictions.The paper identifies these as two practical challenges for using depth and pseudo-labels in unsupervised training.Source: §1 · p. 2
  • The DSU framework iteratively refines pseudo-labels using saliency- and non-saliency-guided depth, while attentive training re-weights samples to emphasize more reliable labels.DSU is not engaged at test time, when the trained model uses only RGB input.Source: §1 · p. 2
  • 35 FPS, with the approach outperforming state-of-the-art unsupervised methods on four public benchmarks; Figure 1 also shows a large-margin NLPR baseline improvement without added inference cost.The paper also reports that test-time speed is much faster than existing unsupervised RGB-D methods and at least on par with the fastest RGB counterparts.Source: §1 · p. 2, Figure 1 · p. 2
  • The approach can also be adapted to fully supervised RGB-D SOD, where adding DSU notably improves existing models.The paper attributes the improvement to incorporating the proposed DSU module.Source: §1 · p. 2

2 RELATED WORK

Earlier RGB-D saliency methods used handcrafted RGB features and depth cues, while deep unsupervised RGB-based SOD learned representations from pseudo-labels. This work proposes refining those labels with disentangled depth information.

2 RELATED WORK

  • Depth-integrating SOD methods address cluttered or low-contrast backgrounds but typically require labor-intensive annotations, motivating unsupervised alternatives.The related-work discussion also notes that handcrafted-prior unsupervised RGB-D methods tend to have inferior performance.Source: §2 · p. 3
  • Traditional RGB-D saliency methods use manually crafted RGB features and depth cues, with prior-based approaches tending to have inferior performance.Examples include center-surround and depth priors, normalized depth and surface-orientation priors, and local background enclosure.Source: §2 · p. 3
  • Deep unsupervised RGB-based SOD methods use traditional-model outputs as pseudo-labels, with prior work fusing model outputs, modeling label noise, or iteratively refining labels.Deep unsupervised learning has also been applied to co-saliency using fusion learning or dedicated loss terms.Source: §2 · p. 3
  • The proposed approach refines pseudo-labels with disentangled depth information, unlike deep RGB-based methods that use handcrafted-method labels directly or self-supervision-based refinement.The passage presents this as the first principled investigation of deep unsupervised RGB-D saliency detection.Source: §2 · p. 3

3 METHODOLOGY

The method iteratively refines pseudo-labels using depth cues and trains the saliency network with attentive weighting for noisy examples. Its depth-disentangled components support label updates during training, while inference uses only the RGB saliency path.

3.1 OVERVIEW

  • DSU combines a saliency network, depth and depth-disentangled networks, and a label-update module to iteratively refine pseudo-labels for saliency-network supervision.The inference stage uses only the black-dashed portion of the architecture.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2.1 DEPTH-DISENTANGLED NETWORK

  • The depth-disentangled network decomposes depth features into saliency-guided and non-saliency-guided representations, trained to capture saliency cues and preserve coherence with the original depth features.It uses MSE supervision and a consistency loss; the reported λ is 0.02.Source: §3.2.1 · pp. 4–5

3.2.2 DEPTH-DISENTANGLED LABEL UPDATE

  • DLU adds saliency-guided depth to coarse saliency predictions and subtracts non-saliency-guided depth, then thresholds, normalizes, and applies CRF to produce updated pseudo-labels.The authors report noticeable improvement after several iterations.Source: §3.2.2 · p. 5

3.3 ATTENTIVE TRAINING STRATEGY

  • ATS alternates between uniform sample weighting to learn general representations and attentive weighting that reduces the weight of higher-loss, more ambiguous examples.At each training-round end, DLU updates pseudo-labels, and the depth masks are also updated using the improved saliency prediction.Source: §3.3 · p. 6
  • With DSU and ATS, the saliency network achieves greater error reduction than the backbone using uniform or attentive weighting alone, and their combination yields better results.Figure 4 evaluates ATS on the NJUD test set; the backbone is trained with initial pseudo-labels.Source: §3.3 · p. 6, Figure 4 · p. 6

4 EXPERIMENTS AND ANALYSES

Experiments compare the approach with unsupervised methods on four RGB-D benchmarks and report stronger results in both RGB-D and RGB-only settings. Analyses examine its components, pseudo-label refinement, and use in fully supervised models.

4.1 DATASETS AND EVALUATION METRICS

  • Four benchmarks are evaluated using F-measure, weighted F-measure, E-measure, MAE, and inference speed, with training samples drawn from NJUD and NLPR.The comparison includes 15 unsupervised methods; RGB-based methods are marked separately for reference.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 COMPARISON WITH THE STATE-OF-THE-ARTS

  • The approach outperforms state-of-the-art unsupervised methods in RGB-D and RGB-only settings, with a lightweight inference design that adds no depth input or computational cost.Table 1 presents the quantitative comparison; Figure 5 provides qualitative comparisons.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 ANALYSIS OF THE EMPIRICAL RESULTS

  • Adding ATS and DSU improves results over the backbone, with full DSU improving F-measure by 11.3% and 23.5% and reducing MAE by 22.4% and 41.9% across the benchmarks.Removing the consistency loss or HA operation degrades performance, and CRF-only refinement performs much worse.Source: §4.3 · pp. 7–8, Table 2 · p. 8
  • ATS improves test-error reduction over uniform or instance-attentive weighting, while alternation intervals that are too large or small yield inferior performance.The comparison also examines ATS combined with DSU across different alternation intervals τ.Source: §4.3 · p. 8, Table 3 · p. 8
  • Pseudo-label quality improves significantly in the first two update rounds and then stabilizes, with updated labels reported as close to ground truth.Figure 6 shows initial labels, labels after CRF, two update rounds, and ground truth for reference.Source: §4.3 · p. 8, Table 4 · p. 8, Figure 6 · p. 8
  • Compared with direct CRF refinement, DSU provides more reliable pseudo-labels; raw depth, with or without Otsu thresholding, can produce poor results.The authors attribute the raw-depth difficulty to large depth variations and foreground objects sharing similar depths with nearby background.Source: Table 5 · p. 8, §4.3 · p. 9
  • Removing the DNonSal branch or directly using raw disentangled depth labels performs worse than the original DSU design.The authors link raw-label performance partly to depth-value variation and intrinsic label noise; learned depth representations can eliminate potential biases in a single raw label.Source: §4.3 · p. 9, Table 6 · p. 9

4.4 APPLICATION TO FULLY-SUPERVISED SETTING

  • A DSU variant consistently improves four existing fully supervised models on two benchmarks, reducing their average NJUD MAE by 18.0%.The adaptation redefines depth labels using ground-truth saliency, then engages DLU to obtain the final improved saliency.Source: §4.4 · p. 9, Table 7 · p. 9

5 CONCLUSION AND OUTLOOK

The paper introduces deep unsupervised RGB-D saliency detection, using iterative pseudo-label refinement and attentive training to address noisy labels. Experiments report strong performance and real-time efficiency; extending the approach to partial labels is future work.

5 CONCLUSION AND OUTLOOK

  • The paper tackles deep unsupervised RGB-D saliency detection by internally engaging and refining pseudo-labels.The DSU iteratively fine-tunes pseudo-labels, while ATS addresses noisy pseudo-labels.Source: §5 · p. 9
  • The authors report superior performance and real-time efficiency for their approach in unsupervised RGB-D saliency detection.Source: §5 · p. 9
  • Extending the approach to scenarios involving partial labels is identified as future work.Source: §5 · p. 9

A APPENDIX

The appendix outlines supplementary metric explanations, experiments, and implementation details. It also identifies potential limitations and directions for future exploration.

A APPENDIX

  • Appendix A.1 elaborates on the evaluation metrics used in the paper.Source: A APPENDIX · p. 16
  • Appendix A.2 covers DSU structures and additional unsupervised and fully supervised experiments, while A.3 gives implementation details.Source: A APPENDIX · p. 16
  • The appendix discusses potential limitations and further research directions in A.4–A.6.Source: A APPENDIX · p. 16

A.1 EVALUATION METRICS

The appendix describes how the paper's saliency metrics assess prediction quality. F-measure balances precision and recall, while MAE measures average absolute error against ground truth.

A.1 EVALUATION METRICS

  • F-measure is the weighted harmonic mean of precision and recall, with β² set to 0.3 to emphasize precision.Source: §A.1 · p. 16
  • MAE is the average absolute difference between a predicted saliency map and ground truth; lower MAE is better.Source: §A.1 · p. 16
  • E-measure captures image-level statistics and local pixel matching, using an enhanced-alignment matrix that reflects correlation after subtracting global means.Source: §A.1 · p. 16

A.2 MORE ANALYSES ON THE PROPOSED DSU

The analyses compare DSU with unsupervised and fully supervised RGB-D SOD methods, examine pseudo-label initialization, and describe the depth-disentangled CNN structures.

Detailed structures

  • The D-Sal and D-NonSal CNN structures are detailed in Figure 7, and the described Conv 3×3 layer uses a 3×3 kernel.BN denotes batch normalization.Source: §A.2 · p. 16, Figure 7 · p. 16

Visual results for unsupervised RGB-D SOD models

  • Figure 8 presents qualitative comparisons with unsupervised RGB-D SOD models, and the authors report predictions close to the true label.The figure caption identifies GT as ground-truth.Source: §A.2 · p. 16

Comparison with current state-of-the-art RGBD SOD methods under fully-supervised scenario

  • Numerical results in Table 8 and visual results in Fig. 9 consistently show the superiority and scalability of the fully supervised DSU variant against 25 existing RGB-D saliency models.The comparison uses the same feature-extraction backbone as DCF.Source: §A.2 · p. 16

Pseudo-label Initialization

  • 0.167 vs. 0.146 on NJUD, 0.119 vs. 0.084 on NLPR, 0.142 vs. 0.119 on STERE, and 0.146 vs. 0.128 on DUTLF-Depth, using MAE metric, comparing backbone with ours.The study also tests alternative initial pseudo-labels, including DCMC.Source: §A.2 · p. 16

A.3 IMPLEMENTATION DETAILS

The unsupervised pipeline uses CPD with a ResNet-50 encoder and fixed image resizing, optimizer, learning rate, and batch size; inference uses RGB alone.

Network and image setup

  • The saliency and depth networks use the CPD feature-extraction backbone with a ResNet-50 encoder pretrained on ImageNet.Training and testing images are resized to 352 × 352.Source: §A.3 · p. 17

Training settings

  • Training uses Adam, a learning rate of 1 × 10−4, and mini-batches of 10, with attentive-training alternation every 3 epochs.The alternation interval amounts to 2τ = 6 epochs in a training round.Source: §A.3 · p. 17

Training and inference

  • The pipeline is trained without human annotations using CDCP outputs as initial pseudo-labels, then predicts saliency from RGB images without depth at inference.Source: §A.3 · p. 17

A.4 LIMITATION AND DISCUSSION

The discussion notes that missing accurate pixel-level annotations limit fine-detail detection, particularly salient-object edges, and presents qualitative comparisons for both evaluation settings.

Limitation and discussion

  • The unsupervised approach does not comprehensively detect fine-grained salient-object details, especially edges, due to the lack of accurate pixel-level annotations.The authors suggest an auxiliary edge constraint, with edge maps generated unsupervisedly by the classical Canny operator.Source: §A.4 · p. 17

Unsupervised qualitative comparison

  • Figure 8 provides qualitative comparisons between the proposed pipeline and unsupervised RGB-D SOD models.The caption identifies GT as ground-truth.Source: Figure 8 · p. 17

Quantitative comparison scope

  • Table 8 reports quantitative results for fully supervised RGB-D saliency methods, using the setup adopted by Piao et al. for DUTLF-Depth.The passage describes this setup as supporting a fair comparison.Source: Table 8 · p. 18

Fully-supervised qualitative comparison

  • Figure 9 compares fully supervised saliency maps, and the authors report more appealing maps than those from the listed state-of-the-art methods.The compared methods include DCF, DSA2F, HAINet, BBSNet, S2MA, UCNet, and FRDT.Source: Figure 9 · p. 18

A.5 ADDITIONAL ANALYSIS

The additional analysis contrasts RGB-only inference with adding depth, reporting a performance-versus-computation trade-off for the latter.

RGB-only inference

  • The DSU design does not add test-time cost and does not require depth maps during inference when using the RGB stream alone.Table 9 compares RGB-only inference with simultaneous RGB and depth input.Source: §A.5 · p. 19, Table 9 · p. 19

RGB and depth inference

  • Adding depth information during inference can further improve model performance, while reducing inference speed and increasing FLOPs and parameter count.Source: §A.5 · p. 19, Table 9 · p. 19

Inference comparison

  • Table 9 compares model performance, inference speed, FLOPs, and parameter number for RGB-only versus RGB-and-depth inference.Source: Table 9 · p. 19, §A.5 · p. 19

A.6 FUTURE DIRECTION

The paper outlines five future directions for deep unsupervised RGB-D SOD, spanning depth use, noisy pseudo-labels, finer disentanglement, and downstream applications.

A.6 FUTURE DIRECTION

  • The work presents deep unsupervised RGB-D SOD as an early effort that avoids human annotations during training.The authors summarize five potential research directions for the area.Source: §A.6 · pp. 19–20

Exploring on the utilization of depth

  • Smoothing inconsistent depth before DLU yielded higher performance, motivating dedicated methods to make depth-revealed saliency regions more uniform.The current approach also uses a depth-disentangled network for object-background variation, thresholding and normalization for within-object variation, and CRF smoothing in DLU.Source: §A.6 · p. 19, Table 10 · p. 19

Exploring on alleviating noise in unsupervised learning

  • Future work could model uncertain regions and use partial BCE loss to reduce the effect of noisy pseudo-labels during training.The proposed attentive training strategy already reweights ambiguous pseudo-labels to alleviate noise.Source: §A.6 · p. 19

Exploring on generating high-quality pseudo-labels

  • The authors propose exploring contrastive loss or partially labeled scribbles to generate higher-quality pseudo-labels.Contrastive pairs could use superpixel candidates with ROI pooling to enlarge foreground-background feature distance.Source: §A.6 · p. 19

Exploring on more fine-grained disentangled framework

  • A finer disentangled framework could add an uncertainty map for “not sure” regions to mitigate noise from uncertain pseudo-labels.The current framework separates saliency-guided and non-saliency-guided depth, reflecting class-agnostic binary segmentation.Source: §A.6 · pp. 19–20

Exploring on more applications

  • The authors identify downstream uses for unsupervised RGB-D SOD, including as a prior for weakly supervised semantic segmentation and action recognition.They also cite image classification, light-field data, medical image processing, and video analysis as application areas for SOD.Source: §A.6 · p. 20
Loading 2205.07179v1…