Source-linked AI summary
UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders
Jing Zhang, Deng-Ping Fan, Yuchao Dai, Saeed Anwar, Fatemeh Sadat Saleh, Tong Zhang, Nick Barnes
TL;DR
Existing RGB-D saliency methods produce a single deterministic saliency map, leaving uncertainty in human annotations under-modeled. UC-Net uses a CVAE-based probabilistic network to generate multiple predictions and saliency consensus to derive an accurate map. Evaluations on six challenging benchmark datasets demonstrate effectiveness in learning saliency-map distributions and establish a new state-of-the-art in RGB-D saliency detection.
Problem
Existing RGB-D saliency methods treat detection as point estimation and produce a single map, failing to capture uncertainty in human GT labeling.
Method
UC-Net uses a conditional variational autoencoder conditioned on RGB-D inputs to model labeling variants and generate multiple saliency predictions.
Results
Evaluations on six standard and challenging benchmark datasets demonstrated UC-Net's superiority in learning the distribution of saliency maps.
Takeaways & Limitations
Saliency consensus enables UC-Net to generate labels discarded during GT annotation and obtain an accurate saliency map from multiple predictions.
Abstract
from arXiv · showhide
In this paper, we propose the first framework (UCNet) to employ uncertainty for RGB-D saliency detection by learning from the data labeling process. Existing RGB-D saliency detection methods treat the saliency detection task as a point estimation problem, and produce a single saliency map following a deterministic learning pipeline. Inspired by the saliency data labeling process, we propose probabilistic RGB-D saliency detection network via conditional variational autoencoders to model human annotation uncertainty and generate multiple saliency maps for each input image by sampling in the latent space. With the proposed saliency consensus process, we are able to generate an accurate saliency map based on these multiple predictions. Quantitative and qualitative evaluations on six challenging benchmark datasets against 18 competing algorithms demonstrate the effectiveness of our approach in learning the distribution of saliency maps, leading to a new state-of-the-art in RGB-D saliency detection.
1. Introduction
RGB-D saliency detection commonly uses deterministic point estimation, which cannot represent uncertainty in human saliency annotations. UC-Net addresses this with probabilistic predictions, saliency consensus, and depth correction.
- Observed behavior: UC-Net produces consistent predictions for single-object images and diverse predictions when multiple salient objects exist.These behaviors are illustrated in the first and second rows of Figure 1.
- Motivation: Existing RGB-D methods map each RGB-D pair to a single GT-based saliency map through deterministic learning.The GT maps are obtained through human consensus or dataset creators.
- Motivation: Human labeling is subjective, so point estimation fails to capture stochastic saliency and can produce partisan predictions.The paper motivates distribution estimation over multiple saliency predictions instead of a single output.
- Approach: UC-Net uses a conditional variational autoencoder to model annotation uncertainty and generate multiple saliency predictions conditioned on RGB-D inputs.The model learns the output-space distribution as a generative model.
- Approach: A hide-and-seek strategy iteratively hides salient foreground from RGB images to obtain diverse predictions despite datasets providing only one GT map.This strategy is introduced to produce diverse and accurate predictions for each input pair.
- Approach: The saliency consensus module mimics majority voting for GT generation, while depth correction produces semantically and geometrically richer depth images to reduce noise.The two components address label-generation uncertainty and noisy depth data, respectively.
2. Related Work
Prior RGB-D saliency work mainly differs in how RGB and depth information are fused, while probabilistic autoencoders provide a related modeling framework. UC-Net applies CVAE to labeling variants and human annotation uncertainty in saliency prediction.
- RGB-D fusion: RGB-D saliency models are commonly categorized as early-fusion, late-fusion, or cross-level fusion approaches.These categories describe how complementary RGB and depth information is combined.
- RGB-D fusion: Examples include superpixel-level early fusion, adaptive late fusion, and multi-scale or complementary-aware cross-level fusion.The cited methods use different structures to combine modality information.
- Probabilistic models: VAE and CVAE use reconstruction and regularization, with CVAE conditioning latent representations on input variables.The KL divergence regularizes the auxiliary posterior toward the conditional prior.
- Novelty: Before UC-Net, CVAE had not been exploited in saliency detection to model human labeling uncertainty.A prior VAE-based saliency method modeled image background rather than labeling variants.
3. Our Model
UC-Net models RGB-D saliency as a distribution of possible saliency maps rather than a single deterministic prediction. Its pipeline combines CVAE-based latent sampling, depth refinement, feature fusion, and saliency consensus to produce final predictions.
- 3.3. Saliency Consensus Module: During testing, repeated PriorNet sampling generates multiple predictions, which saliency consensus aggregates through majority voting.The consensus process follows the mechanism used to generate ground-truth saliency maps.
- 3.1. Probabilistic RGB-D Saliency Model via CVAE: LatentNet produces low-dimensional Gaussian latent variables that represent stochastic labeling variants for saliency prediction.PosteriorNet is used during training, and PriorNet is sampled during testing.
- 3.1. Probabilistic RGB-D Saliency Model via CVAE: UC-Net uses a conditional variational autoencoder to learn the distribution of saliency maps conditioned on RGB-D inputs.PriorNet uses RGB-D input, while PosteriorNet additionally uses the ground-truth saliency map during training.
- 3.1. Probabilistic RGB-D Saliency Model via CVAE: The hiding strategy iteratively removes predicted salient regions to generate diverse training annotations from datasets containing only one ground-truth map.After repeated hiding, corresponding ground truths provide alternative annotations for training the posterior and prior distributions.
- 3.1. Probabilistic RGB-D Saliency Model via CVAE: PredictionNet mixes stochastic and deterministic features channel-wise before producing a saliency map.This design prevents the network from relying only on deterministic SaliencyNet features and preserves latent labeling variation.
- 3.2. DepthCorrectionNet: DepthCorrectionNet refines noisy depth using semantic guidance and a boundary-alignment regularizer with RGB intensity edges.Its loss combines smooth ℓ1 agreement with raw depth and boundary IOU alignment.
4. Experimental Results
UC-Net is evaluated against 18 methods on six RGB-D datasets, with experiments covering benchmark performance, prediction diversity, latent-space scale, depth representation, and label generation. It consistently achieves strong benchmark results while producing diverse predictions and benefiting from depth correction and augmented labels.
- Performance Comparison: Experiments compare UC-Net with 18 algorithms across six RGB-D saliency datasets using MAE, mean F-measure, mean E-measure, and S-measure.The datasets include NJU2K, NLPR, SSB, LFSD, DES, and SIP.
- Performance Comparison: UC-Net consistently achieves the best performance across all datasets, with especially strong gains on SSB and SIP.The reported improvements include higher S-measure, E-measure, and F-measure and substantially lower MAE.
- Performance Comparison: UC-Net produces stable E-measure and F-measure curves while maintaining the best performance among competing methods.Figure 7 reports these curves on four testing datasets.
- Qualitative Comparisons: UC-Net produces accurate predictions and diverse outputs for complex-background images, unlike M-head and MC-dropout, which yield consistent predictions for ambiguous images.Figure 8 compares DMRA, M-head, MC-dropout, and two UC-Net samples.
- Ablation Study: Setting the latent-space scale to K = 8 performs best, while predictions remain relatively stable for K ∈[6, 10].Using K = 32 produces worse results than the reported UC-Net configuration.
- Ablation Study: Removing DepthCorrectionNet worsens performance, while the proposed solution improves DES results by around 4% on S-measure, E-measure, and F-measure.The ablation directly concatenates RGB and depth inputs instead of correcting depth first.
- Ablation Study: The CVAE-based UC-Net outperforms M-head and MC-dropout variants, although both alternatives also outperform some state-of-the-art models.The M-head and MC-dropout variants are evaluated as M5 and M6, respectively.
- Ablation Study: Augmented training data improves performance over the provided training data, supporting the effectiveness of the new label-generation technique.The comparison is reported between M9 and M8.
5. Conclusion
The paper presents UC-Net as an uncertainty-aware RGB-D saliency detector that learns a distribution of saliency maps rather than a single point estimate. Across six challenging benchmarks, the approach demonstrates effectiveness and is positioned for extension to other saliency and segmentation tasks.
- Conclusion: UC-Net uses a conditional variational autoencoder to learn saliency-map distributions and generate multiple labels through saliency consensus.This addresses uncertainty discarded when ground-truth annotations are consolidated.
- Conclusion: Quantitative and qualitative evaluations on six standard, challenging benchmarks demonstrate the superiority of UC-Net in learning saliency-map distributions.
- Future Work: The authors propose extending the approach to video, RGB, and co-saliency detection and collecting multiple-annotation datasets for other segmentation problems.The cited future directions include VSOD, RGB SOD, Co-SOD, interactive image segmentation, and camouflaged object detection.