Uncertainty Inspired RGB-D Saliency Detection

Jing Zhang; Deng-Ping Fan; Yuchao Dai; Saeed Anwar; Fatemeh Saleh; Sadegh Aliakbarian; Nick Barnes

Source-linked AI summary

Uncertainty Inspired RGB-D Saliency Detection

Jing Zhang, Deng-Ping Fan, Yuchao Dai, Saeed Anwar, Fatemeh Saleh, Sadegh Aliakbarian, Nick Barnes

arXiv:2009.03075v1Sep 7, 2020cs.CV

TL;DR

Existing RGB-D saliency models make deterministic point predictions despite uncertainty in human annotations. This paper introduces a latent-variable generative framework with CVAE and ABP inference to produce stochastic saliency maps, and reports state-of-the-art performance on RGB-D benchmarks. The two inference schemes both model output distributions, but ABP requires longer training while CVAE adds parameters.

Problem
Existing RGB-D saliency models treat detection as point estimation and do not capture uncertainty in ground-truth labeling.
Method
A generator maps RGB-D input and latent variable z to conditional saliency predictions, while CVAE or ABP inference estimates z from an approximate or true posterior.
Results
The approach generates multiple saliency predictions and achieves state-of-the-art performance on six RGB-D benchmark datasets.
Takeaways & Limitations
Modeling saliency as a distribution represents human annotation uncertainty and supports diverse predictions rather than a single output.
Takeaways & Limitations
ABP requires several Langevin-dynamics MCMC steps, leading to relatively longer training time, while CVAE uses an extra encoder and more parameters.

Abstract

from arXiv · show

We propose the first stochastic framework to employ uncertainty for RGB-D saliency detection by learning from the data labeling process. Existing RGB-D saliency detection models treat this task as a point estimation problem by predicting a single saliency map following a deterministic learning pipeline. We argue that, however, the deterministic solution is relatively ill-posed. Inspired by the saliency data labeling process, we propose a generative architecture to achieve probabilistic RGB-D saliency detection which utilizes a latent variable to model the labeling variations. Our framework includes two main models: 1) a generator model, which maps the input image and latent variable to stochastic saliency prediction, and 2) an inference model, which gradually updates the latent variable by sampling it from the true or approximate posterior distribution. The generator model is an encoder-decoder saliency network. To infer the latent variable, we introduce two different solutions: i) a Conditional Variational Auto-encoder with an extra encoder to approximate the posterior distribution of the latent variable; and ii) an Alternating Back-Propagation technique, which directly samples the latent variable from the true posterior distribution. Qualitative and quantitative results on six challenging RGB-D benchmark datasets show our approach's superior performance in learning the distribution of saliency maps. The source code is publicly available via our project page: https://github.com/JingZhang617/UCNet.

1 INTRODUCTION

RGB-D saliency detection is commonly posed as deterministic point estimation, although human saliency labeling is subjective and can vary across annotators. The paper therefore introduces probabilistic prediction with latent uncertainty and two inference strategies for generating diverse saliency maps.

Motivation: Conventional RGB-D models learn a single saliency map from RGB-D input and ground-truth annotations as point estimation.They model the mapping as Y = f(X; θ).
Motivation: Human annotators may disagree about salient foreground regions, so deterministic ground-truth mapping fails to capture labeling uncertainty.The same region may be labeled foreground by one annotator and background by another.
Proposed framework: The proposed generative architecture introduces latent variable z to model annotation uncertainty and maps RGB-D input plus z to stochastic saliency predictions.Its inference model progressively refreshes z.
Prediction diversity: Prediction diversity depends on image context: simple scenes yield consistent outputs, whereas complex scenes can yield diverse saliency maps.The figure illustrates this contrast between simple and uncertain salient regions.
Inference strategies: The framework infers z with either a CVAE encoder that approximates its posterior or ABP that samples from the true posterior using Langevin-dynamics MCMC.Both schemes can produce diverse predictions during testing.

2 RELATED WORK

Prior work covers RGB-D fusion and deep probabilistic models, but generative models had not been used to model annotation uncertainty in saliency detection beyond the authors’ preliminary work. This paper extends that direction with CVAE and ABP inference, where ABP directly samples latent variables from the true posterior.

RGB-D saliency models: Existing RGB-D saliency models are categorized by how RGB and depth features are fused: early, late, or cross-level fusion.These approaches seek complementary information between the two modalities.
VAE or CVAE-based models: VAE and CVAE models combine encoder-decoder architectures with reconstruction and distribution-regularization objectives for probabilistic vision tasks.CVAE conditions Gaussian latent variables on the input observation.
GAN-based dense models: GAN-based dense prediction methods use adversarial objectives for supervised structural regularization or semi-supervised guidance.They include applications to saliency-map discrimination and domain adaptation.
Uniqueness of the proposed solutions: Generative models had not been exploited to model annotation uncertainty in saliency detection apart from the authors’ preliminary version.The present work adds an MCMC-based alternative to its earlier CVAE solution.
Uniqueness of the proposed solutions: CVAE inference approximates the posterior with an extra encoder and may suffer posterior collapse, whereas ABP uses gradient-based MCMC to sample latent variables directly from the true posterior.The paper reports that both approaches produce stochastic saliency predictions.

3 OUR MODEL

The model learns a conditional distribution of RGB-D saliency maps by combining a latent-variable generator with inference models that update the latent representation. It supports CVAE-based posterior approximation and ABP-based direct posterior sampling, with losses and consensus estimation designed for structured stochastic predictions.

Probabilistic RGB-D saliency model: The framework models Pω(Y|X,z), using latent variable z to represent uncertainty in salient regions and a generator that maps RGB-D input X and z to stochastic predictions.The generator uses a ResNet50 backbone, DenseASPP-style receptive-field enlargement, top-down feature aggregation, and residual channel attention.
Loss function: The hybrid objective combines CVAE and GSNN losses as LHybrid = αLCVAE + (1 −α)LGSNN to balance their objectives.The CVAE loss includes reconstruction or negative log-likelihood terms and KL regularization; the structure-aware loss penalizes incorrect predictions near object boundaries.
CVAE inference: CVAE inference uses PosteriorNet Qφ(z|X,Y) during training and PriorNet Pθ(z|X) for prior sampling, with KL divergence regularizing their distributions.The CVAE likelihood models Pω(Y|X,z), while linear KL annealing weights the divergence term to help prevent posterior collapse.
ABP inference: ABP inference directly samples z from the true posterior through gradient-based Langevin-dynamics MCMC instead of approximating it with an encoder.The paper motivates this alternative by noting that CVAE posterior collapse can make z independent of prediction Y and unable to encode annotation uncertainty.
Output estimation: Multiple stochastic predictions can be averaged, summarized through the mean latent variable, or aggregated with a Saliency Consensus Module inspired by majority voting among annotators.The consensus module is enabled by distributional prediction, unlike a single deterministic saliency map that cannot undergo majority voting.

4 EXPERIMENTAL RESULTS

Experiments evaluate the method on six RGB-D datasets using four saliency metrics, comparisons with conventional and deep models, qualitative examples, stochastic outputs, and ablations. The proposed CVAE and ABP variants achieve strong benchmark performance while producing diverse predictions and maintaining comparable inference time.

Experimental setup: Experiments use six RGB-D datasets and compare against 18 methods with MAE, F-measure, S-measure, and E-measure.The evaluation includes five established datasets and the SIP dataset, alongside ten handcrafted and eight deep competing methods.
Quantitative comparison: The CVAE and ABP variants consistently achieve the best performance across the six datasets.CVAE denotes the framework using posterior approximation, while ABP updates the latent variable through alternating back-propagation.
Quantitative comparison: On SSB and SIP, the method improves S-measure, E-measure, and F-measure by about 2.5% and reduces MAE by 1.5% versus deep models.These comparisons are reported in Table 1 against the listed “Deep Models” baseline.
Qualitative comparison: The method produces stable, best-performing E-measure and F-measure curves and qualitative predictions that handle salient objects in complex backgrounds.The qualitative comparisons include large and small salient objects as well as backgrounds sharing color and texture with the foreground.
Probabilistic evaluation: The probabilistic model achieves inference time comparable to competing methods while generating distributions of plausible saliency maps.The distribution evaluation distinguishes consistent predictions for simple images from diverse predictions for complex images.
Structured output generation: Both inference models generate reasonable stochastic predictions, while the Saliency Consensus Module produces deterministic predictions consistent with the ground truth.The stochastic samples represent plausible outputs from the latent-variable model, and the consensus output follows the evaluation pipeline.
Ablation studies: Ablations show that the proposed hybrid loss outperforms using only the CVAE loss or GSNN loss, while varying latent dimension yields relatively stable performance.The latent dimension is tested from 3 to 32, and the hybrid formulation remains stronger than either individual loss.

5 CONCLUSION

The paper frames RGB-D saliency detection as probabilistic prediction of annotation uncertainty rather than deterministic point estimation. Its generative model uses CVAE or ABP inference to produce multiple saliency maps and consensus predictions, with experiments demonstrating strong distribution-learning performance.

Conclusion: The proposed model learns a distribution of saliency maps by introducing a latent variable for uncertainty in human annotations.This differs from methods that predict a single saliency map through point estimation.
Conclusion: CVAE approximates the latent-variable posterior, whereas ABP samples it directly using gradient-based MCMC.Both inference schemes are used to learn the same generative saliency framework.
Conclusion: The model generates multiple predictions and uses a Saliency Consensus Module to produce accurate deterministic saliency maps.The consensus stage follows a process analogous to ground-truth annotation generation.
Conclusion: Quantitative and qualitative evaluations on six challenging RGB-D benchmarks demonstrate the approach’s superiority in learning saliency-map distributions.The paper also extends the framework to RGB saliency detection without changing the network structure apart from the input modality.
Conclusion: CVAE uses more network parameters, whereas ABP requires longer training because it repeatedly updates the latent variable with Langevin-dynamics MCMC.The two inference strategies therefore involve different computational trade-offs.