Lightweight Probabilistic Deep Networks

Jochen Gast; Stefan Roth

Source-linked AI summary

Lightweight Probabilistic Deep Networks

Jochen Gast, Stefan Roth

arXiv:1805.11327v1May 29, 2018cs.CVcs.LGstat.ML

TL;DR

Probabilistic deep networks offer useful reliability information, but existing Bayesian and sampling-based approaches can be too costly for practical CNN use. The paper introduces lightweight probabilistic output layers and assumed-density-filtering propagation, finding retained predictive performance and efficiency, uncertainty correlated with empirical error, and increased adversarial robustness.

Problem
Deep networks usually lack explicit, well-calibrated uncertainty representations, while Bayesian and sampling approaches can make inference costly.
Method
The paper uses probabilistic output layers for classification and regression and assumed density filtering to propagate activation distributions through networks.
Results
The probabilistic networks retain deterministic counterparts’ predictive power and efficiency, produce uncertainty correlated with empirical error, and improve classification robustness to adversarial attacks.
Takeaways & Limitations
These models provide efficient drop-in replacements for vision networks that can assess prediction uncertainty without requiring full Bayesian inference.
Takeaways & Limitations
The approach disregards model uncertainty and targets aleatoric, heteroscedastic uncertainty.

Abstract

from arXiv · show

Even though probabilistic treatments of neural networks have a long history, they have not found widespread use in practice. Sampling approaches are often too slow already for simple networks. The size of the inputs and the depth of typical CNN architectures in computer vision only compound this problem. Uncertainty in neural networks has thus been largely ignored in practice, despite the fact that it may provide important information about the reliability of predictions and the inner workings of the network. In this paper, we introduce two lightweight approaches to making supervised learning with probabilistic deep networks practical: First, we suggest probabilistic output layers for classification and regression that require only minimal changes to existing networks. Second, we employ assumed density filtering and show that activation uncertainties can be propagated in a practical fashion through the entire network, again with minor changes. Both probabilistic networks retain the predictive power of the deterministic counterpart, but yield uncertainties that correlate well with the empirical error induced by their predictions. Moreover, the robustness to adversarial examples is significantly increased.

1. Introduction

The paper targets unreliable, overconfident, and computationally costly uncertainty estimation in deep networks by introducing lightweight probabilistic outputs and activation propagation. These approaches retain predictive performance and efficiency while providing useful uncertainty estimates and greater adversarial robustness.

Deep CNNs typically represent activations and predictions as point estimates, leaving prediction reliability without an explicit uncertainty representation.
Uncertainty is especially important in settings such as medical applications and autonomous driving, where highly uncertain predictions can receive particular care.
The paper proposes probabilistic output layers that make minimal changes to existing networks while keeping inference fast and efficient.
Assumed density filtering propagates activation uncertainties through the network in a single pass without significantly increasing the number of parameters.
Probabilistic output and activation-propagation networks retain deterministic counterparts’ predictive power and efficiency while improving uncertainty assessment and adversarial resilience.

2. Related Work

Prior probabilistic neural-network approaches include generative models, Bayesian parameter treatments, sampling methods, and uncertainty-propagation architectures. The paper distinguishes its lightweight supervised-learning approach by avoiding probabilistic parameters and extensive test-time sampling while focusing on practical vision networks.

Earlier probabilistic neural networks often modeled latent generative representations, whereas this work addresses supervised vision tasks and repurposes Gaussian layer transformations for activation uncertainty.
Bayesian parameter treatments provide principled model uncertainty but increase parameter counts and test-time computational burden, which this work avoids.
Sampling-based uncertainty methods can require multiple networks or test-time Monte Carlo estimates, limiting practicality for large vision architectures.
The paper focuses on practical, fast aleatoric heteroscedastic uncertainty and disregards model uncertainty.
Some uncertainty-propagation methods rely on assumptions that are difficult to generalize across nonlinear operations such as max pooling.
Unlike approaches that probabilistically model weights and activations with exponential-family distributions, this work avoids Gaussian parameter treatment to remain lightweight.

3. Uncertainty Prediction in Supervised CNNs

The section presents two lightweight ways to add uncertainty to CNNs: probabilistic outputs and activation-uncertainty propagation through intermediate layers.

Overview: Two approaches replace standard CNN prediction with probabilistic outputs or propagate activation uncertainties through the network using ADF.The first changes only the output layer; the second applies uncertainty propagation across intermediate layers.
CNNs with probabilistic outputs: Probabilistic output layers parameterize a predictive distribution instead of directly predicting point outputs.The final layer predicts distribution parameters encoding both the output and its associated uncertainty.
CNNs with probabilistic outputs: Output-layer probabilistic modeling increases parameters only minimally because changes are confined to predicting distribution parameters in the final layer.For a Gaussian output, the final layer predicts mean and variance.
Deep uncertainty propagation using ADF: ADF propagates activation uncertainties by approximating successive layer activations with independent Gaussian distributions in a single forward pass.Each layer transforms the approximate activation distribution, and ADF matches first- and second-order moments.
Deep uncertainty propagation using ADF: The method approximates activation distributions rather than probabilistic weights, yielding a smaller computational footprint while accommodating multimodal activation posteriors approximately.The approximation is simpler than Bayesian approaches that model weight uncertainty.
Variational approximation of common layers: Dense, convolutional, and deconvolutional layers propagate moments in closed form, while pooling and nonlinearities require suitable approximations or derived formulas.For linear layers, means and variances are propagated separately; ReLU causes them to interact.

4. Supervised Probabilistic Training

The section describes maximum conditional likelihood training for probabilistic regression and classification outputs. It introduces power exponential regression outputs and Dirichlet classification outputs to represent predictive uncertainty.

Maximum conditional likelihood learning: Probabilistic networks are trained by maximizing conditional likelihood, implemented numerically as minimizing negative log-likelihood.The objective uses the full predictive distribution rather than only a point estimate.
Maximum conditional likelihood learning: For Gaussian regression, conditional likelihood weights squared error by predicted precision and penalizes excessive predicted variance.The network can reduce loss by improving its mean prediction or increasing variance where it expects larger errors.
Regression with power exponential outputs: Power exponential outputs provide a probabilistic analogue of common vision regression losses, with k = 1/2 yielding a multivariate Laplacian for per-pixel L2-norm objectives.The paper applies this family to optical-flow regression and reports good endpoint-error performance.
Classification with Dirichlet outputs: Softmax probabilities can be confident or uncertain but are not necessarily calibrated, so high entropy does not reliably correlate with actual classification error.The proposed classification output uses a continuous density on the probability simplex.
Classification with Dirichlet outputs: The Dirichlet output represents a simplex location m as the point prediction and a scale s as the density’s spread around that prediction.The scale is derived from neuron uncertainties rather than predicted directly.
Classification with Dirichlet outputs: Dirichlet scale is computed from uncertainty weighted by softmax activation, emphasizing uncertainty associated with influential class activations.The constants c1 and c2 control distribution peakedness and uncertainty amplification.
Classification with Dirichlet outputs: Dirichlet training applies Laplace smoothing to discrete labels because the Dirichlet distribution supports continuous vectors on the unit simplex.The smoothing parameter trades off robustness and accuracy.

5. Experiments

Experiments evaluate lightweight probabilistic FlowNet and classification variants, showing competitive predictive performance, practical inference, uncertainty aligned with errors, and improved adversarial robustness.

5.1. Probabilistic regression with FlowNet: The probabilistic FlowNet variants retain predictive power while adding uncertainty estimates that correlate highly with actual endpoint error.The reported uncertainty indicates where the model fails and succeeds.
5.1. Probabilistic regression with FlowNet: The probabilistic regression models are evaluated on Sintel and a 640-image FlyingChairs hold-out, with execution speed measured on a Nvidia GTX 1080 Ti.Table 1 reports endpoint error and frames per second; Table 2 reports average log likelihoods.
5.1. Probabilistic regression with FlowNet: FlowNetADF outperforms the deterministic counterpart with a 4% improvement on Sintel clean.FlowNetDropOut is an order of magnitude slower than the proposed probabilistic networks.
5.2. Probabilistic classification: The Dirichlet output layer shows a strong correlation between predictive entropy and empirical error, unlike the weak correlation of softmax.Figure 3 compares categorical entropy with data cross-entropy on CIFAR10.
5.2. Probabilistic classification: On CIFAR10 and MNIST, classification performance is similar for networks trained with the same loss, while Dir + CLLH outperforms softmax-based losses.Test-time dropout is slightly better in CIFAR cross-entropy but requires many forward passes.
5.2. Probabilistic classification: The Dirichlet layer is more robust against gradient sign attacks than the softmax layer, while test-time dropout is more attack-prone than Dir + CLLH.Table 4 reports accuracy after attacks at three strengths.

6. Conclusion

The paper presents lightweight probabilistic output layers and uncertainty propagation for supervised deep networks. These models preserve predictive power and most computational efficiency while estimating uncertainty and improving classification robustness to adversarial attacks.

6. Conclusion: The paper proposes probabilistic output layers and assumed-density-filtering-based uncertainty propagation for supervised deep networks.Regression uses a general power exponential layer, while classification uses a Dirichlet layer.
6. Conclusion: The proposed models can serve as efficient drop-in replacements for deep vision networks.They are trained by conditional likelihood maximization.
6. Conclusion: The models retain predictive power and most computational efficiency while producing uncertainty estimates highly correlated with empirical error.Classification networks additionally become more robust against adversarial attacks.

– Supplemental Material –

The supplemental material documents the uncertainty-propagation construction, probabilistic layer details, and visual demonstrations of uncertainty prediction in classification and regression.

Supplemental Material: The supplemental material derives recipes for creating uncertainty-propagating layers.It also provides additional details on the layer types used in the main paper.
Supplemental Material: The supplemental material visualizes uncertainty prediction and illustrates benefits with classification and regression examples.These demonstrations complement the main paper’s experiments.

A. Assumed Density Filtering

The framework applies assumed density filtering to neural-network activation densities, using iterative approximations that match first- and second-order moments under a Gaussian assumption.

A. Assumed Density Filtering: ADF initializes the approximation with the Gaussian input distribution and updates factors across consecutive network layers.The true posterior at each iteration combines the newest factor with previously approximating factors.
A. Assumed Density Filtering: For Gaussian approximations, minimizing the update objective is equivalent to matching the first two moments of the true and approximate joint densities.
A. Assumed Density Filtering: The approximate posterior factorizes by separating an activation variable from the joint distribution of the remaining factors.
A. Assumed Density Filtering: First-moment updates leave activations outside the current layer unchanged and recover the recipe used for the last-layer activations.
A. Assumed Density Filtering: Applying the same derivation to squared activations yields the corresponding variance update.

B. Uncertainty Propagation Layers

The paper implements uncertainty propagation across common neural-network layers by approximating activation distributions and deriving moment updates for linear, pooling, and nonlinear operations.

B. Uncertainty Propagation Layers: Linear layers, including fully connected layers, convolutions, and deconvolutions, are formalized as linear functions for uncertainty propagation.
B. Uncertainty Propagation Layers: Average pooling is represented as a linear layer whose weights implement averaging, with variance scaling determined by the number of inputs.The relevant input count varies with pooling size.
B. Uncertainty Propagation Layers: Max pooling is modeled through the distribution of the maximum of its input variables.
B. Uncertainty Propagation Layers: Non-Gaussian maximum responses are approximated by a univariate normal distribution with analytically derived mean and variance.
B. Uncertainty Propagation Layers: Max pooling with more than two inputs is implemented by folding the operation horizontally and then vertically.
B. Uncertainty Propagation Layers: The regression uncertainty assessment compares predictive differential entropy with average endpoint error.High entropy indicates high variance and, in the illustrated region, coincides with larger endpoint error.

C. Calibration of Regression Uncertainties

The regression experiments evaluate whether predicted uncertainty reflects optical-flow prediction error, finding a strong correspondence between uncertainty and endpoint error.

C. Calibration of Regression Uncertainties: Predicted uncertainties are well correlated with actual endpoint error in probabilistic optical-flow networks.
C. Calibration of Regression Uncertainties: Figure 5 plots FlowNetADF endpoint error against differential entropy for every image in the FlyingChairs test set.
C. Calibration of Regression Uncertainties: The authors conclude that predicted variances can be used to assess where the model succeeds or fails.

D. Exponential Power Outputs

The optical-flow networks use two-dimensional exponential power outputs to represent predictive densities with different uncertainty levels and directional variances.

D. Exponential Power Outputs: Figure 6 illustrates high-variance and low-variance predictions when the mean optical-flow prediction is zero.
D. Exponential Power Outputs: A separate illustrative case shows a predictive density whose horizontal and vertical variances differ by an order of magnitude.
D. Exponential Power Outputs: The exponential power unit represents the predictive distribution at an output node using horizontal and vertical optical-flow components.

E. Additional Examples

Additional examples show that lightweight probabilistic networks preserve useful predictions while expressing uncertainty differently from deterministic softmax outputs. Their uncertainty tends to increase on incorrect classifications and correlates with optical-flow endpoint error.

Dirichlet outputs usually remain less confident than softmax predictions on correctly classified CIFAR10 images.Softmax outputs often have very low entropy, whereas Dirichlet predictions rarely collapse to a single class.
On misclassified CIFAR10 images, Dirichlet predictions are generally less confident and have higher entropy than deterministic softmax predictions.This difference is visible even when both output layers fail on the same examples.
When the Dirichlet layer fails while softmax succeeds, its predictions are often highly uncertain.These cases provide examples where high entropy accompanies an incorrect Dirichlet prediction.
When softmax fails while the Dirichlet layer succeeds, softmax predictions can remain highly confident despite the wrong class.The examples highlight a mismatch between softmax confidence and correctness.
Optical-flow examples from Sintel show that FlowNetADF and FlowNetProbOut provide uncertainties while remaining competitive on endpoint error.The displayed predictions compare uncertainty patterns with optical-flow results across additional Sintel examples.