SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates

1. Introduction

• DNNs can be overconfident or miscalibrated, including on out-of-distribution samples, motivating uncertainty estimates for real-world applications.The cited applications include self-driving cars, cybersecurity, and medical diagnosis.Source: §1 · p. 1
• SDE-Net explicitly separates aleatoric and epistemic uncertainty, avoids specifying parameter priors and inferring posteriors, and applies to classification and regression.The paper contrasts this with Bayesian methods’ inference challenges and non-Bayesian methods’ cost or conflation of uncertainty sources.Source: §1 · pp. 1–2
• SDE-Net represents hidden-state transformations with an SDE, using a drift net for prediction and a diffusion net that encourages high diffusion outside the training distribution.The paper theoretically analyzes existence and uniqueness of the stochastic system’s solution.Source: §1 · p. 2
• SDE-Net outperforms state-of-the-art uncertainty methods or achieves competitive results across out-of-distribution, misclassification, adversarial-sample, and active-learning tasks.The evaluations cover various datasets.Source: §1 · p. 2

2. Aleatoric Uncertainty and Epistemic Uncertainty

• Aleatoric uncertainty is task-inherent randomness, while epistemic uncertainty reflects model ignorance from missing observations and is high where training data are scarce.Source: §2 · p. 2
• Low aleatoric and epistemic uncertainty yields concentrated predictions; high aleatoric uncertainty shifts predictions toward central or large-variance regions, while high epistemic uncertainty disperses them.Source: §2 · p. 2
• Bayesian neural networks model epistemic uncertainty with parameter distributions, but prior choice is difficult and exact posterior inference is often intractable for large networks.Variational inference may underestimate true uncertainty; MCMC can be time-consuming and costly for large networks.Source: §2 · p. 2

3. Uncertainty Quantification via Neural Stochastic Differential Equation

• SDE-Net models hidden-state transformations as stochastic dynamics, separating predictive drift from Brownian-motion diffusion for epistemic uncertainty.The method is designed to distinguish aleatoric and epistemic uncertainty without specifying parameter priors or performing complicated Bayesian inference.Source: §3 · p. 3, §3.2 · p. 3

3.1. Neural Net as Deterministic Dynamical System

• ResNet layer transformations can be viewed as discretizations of a continuous dynamical system, with neural ODEs parameterizing its dynamics (Chen et al., 2018).An ODE solver can evaluate hidden-unit dynamics at arbitrary accuracy, with better memory and parameter efficiency.Source: §3.1 · p. 3

3.2. Modeling Epistemic Uncertainty with Brownian Motion

• Brownian motion makes the hidden-state dynamics stochastic, with larger diffusion variance in data-scarce regions indicating greater epistemic uncertainty.The variance of the final-time solution xT provides an epistemic uncertainty estimate; Figure 2 illustrates lower trajectory variance for small g and greater scatter for large g.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3. SDE-Net for Uncertainty Estimation

• Separate drift and diffusion nets let the drift support predictive accuracy and aleatoric uncertainty while diffusion represents epistemic uncertainty.SDE-Net samples multiple realizations of one stochastic model to estimate both uncertainties, rather than training multiple deterministic networks as ensembles do.Source: §3.3 · pp. 3–4
• The training objective uses out-of-distribution samples that may be generated by adding Gaussian noise to inputs or drawn from a carefully chosen real dataset.SDE-Net shares parameters across layers and simplifies optimization by making diffusion variance depend on the starting point x0 rather than instantaneous xt.Source: §3.3 · p. 4

3.4. Theoretical Analysis

• The theoretical analysis addresses existence and uniqueness, with a stated requirement that both drift and diffusion be uniformly Lipschitz continuous.The diffusion output is capped by a hyperparameter σmax to avoid explosive solutions and unstable optimization.Source: §3.4 · pp. 4–5

3.5. SDE-Net Training

• For efficient training, SDE-Net uses a fixed-step Euler–Maruyama scheme, with the drift and diffusion nets optimized alternately.The authors report that sampling one path per data point during training suffices empirically.Source: §3.5 · p. 5

4.1. Experimental Setup

• For fair comparison, SDE-Net replaces residual blocks with one SDE-Net block of matching hidden-layer count, uses one training path, and performs 10 stochastic test-time passes.Gaussian noise is added to training data as pseudo-OOD data for PN and SDE-Net.Source: §4.1 · p. 5

4.2. Out-of-Distribution Detection

• Across MNIST and SVHN OOD combinations, SDE-Net consistently leads detection, outperforming DeepEnsemble and Prior Network, including when both methods train with Gaussian pseudo-OOD data.DeepEnsemble is the strongest baseline but requires training multiple networks; SDE-Net is also more robust to the choice of OOD training data.Source: Table 1 · p. 6, §4.2 · p. 6
• On Year Prediction MSD versus Boston Housing OOD detection, SDE-Net performs strongly despite weak baselines; AUPR-out is preferred because the test data are imbalanced.The diffusion net models input-dependent epistemic uncertainty, encouraging higher uncertainty for OOD data and lower uncertainty for in-distribution data.Source: Table 2 · p. 7, §4.2 · p. 7

4.3. Misclassification Detection

• For MNIST and SVHN misclassification detection, p-SGLD performs best overall, while SDE-Net is comparable to DeepEnsemble and outperforms other baselines.The authors argue SDE-Net is a practical choice because p-SGLD requires storing parameter copies and DeepEnsemble requires training multiple models.Source: Table 3 · p. 7, §4.3 · p. 7

4.4. Adversarial Sample Detection

• Under PGD, MNIST baseline AUROCs fall below 70% after 60 iterations, while SDE-Net remains above 80% after 100; on SVHN, all methods except DeepEnsemble quickly become overconfident.The authors identify high-dimensional adversarial samples as an area needing efficient, robust uncertainty-aware models.Source: §4.4 · pp. 7–8, Figure 6 · p. 8
• Under FGSM, SDE-Net achieves AUROC similar to p-SGLD and outperforms other methods; on SVHN it reaches 100% at perturbation size one while other baselines reach about 90%.On MNIST, all methods approach 100% AUROC at large perturbation sizes.Source: §4.4 · p. 7, Figure 5 · p. 8

4.5. Active Learning

• On Year Prediction MSD active learning, SDE-Net's RMSE consistently decreases as labels are acquired, while deterministic NN and DeepEnsemble performance degrades after several iterations.BBP and p-SGLD show negligible gains even after 100 acquisitions; the authors attribute degradation to selecting uninformative points and overfitting with little training data.Source: Figure 7 · p. 8, §4.5 · p. 8

Uncertainty estimation

• Bayesian neural nets place distributions over model parameters, but exact posterior inference is often intractable and choosing priors is challenging.Approximation approaches include variational inference, Laplace approximation, expectation propagation, stochastic gradient MCMC, and MC-dropout.Source: §5 · p. 8
• DeepEnsemble trains multiple neural networks and reports competitive uncertainty estimates, while other non-Bayesian methods include entropy penalties, confidence losses, and Prior Networks.Prior Networks model Dirichlet distributions over categorical outputs and are limited to classification tasks.Source: §5 · p. 9

Neural dynamic system

• Neural-network architectures can be viewed as discretizations of ODEs, and neural ODEs extend ResNets to continuous-depth networks with ODE solvers.Prior work focused on predictive accuracy; SDE-Net instead uses an SDE formulation and Brownian motion for uncertainty quantification.Source: §5 · p. 9
• Other neural-SDE work studies connections to infinitely deep residual networks and continuous-time-series modeling, while this work focuses on uncertainty quantification.The authors describe these approaches as developed simultaneously.Source: §5 · p. 9

6. Conclusion

• SDE-Net is proposed as a neural stochastic differential equation model for quantifying uncertainty in deep neural networks.The model separates uncertainty sources and is described as simpler and more straightforward than Bayesian neural nets.Source: §6 · p. 9
• Experiments show strong performance compared with state-of-the-art uncertainty-quantification techniques on classification and regression tasks.The authors characterize the neural-SDE approach as their first study connecting stochastic dynamical systems and neural nets for uncertainty quantification.Source: §6 · p. 9

S. 1. Proof of Theorem 1

• The auxiliary bound is obtained by iterating an inequality: its terms form the series for Ae^Bt, while the remaining iterated-integral term vanishes uniformly.This supplies the bound used in the proof's convergence argument.Source: S. 1. Proof of Theorem 1 · p. 12
• The SDE has at most one solution for each initial value, and the constructed process sequence converges uniformly on compact time intervals to a continuous solution.The uniqueness argument uses the Lipschitz conditions on the drift and diffusion nets; the limit process establishes existence.Source: S. 1. Proof of Theorem 1 · p. 13
• For existence, the proof defines adapted processes recursively and uses the drift net's Lipschitz property to control their convergence.The limiting process has continuous trajectories by uniform convergence on compact intervals.Source: S. 1. Proof of Theorem 1 · p. 13

S. 2.1. Classification Setup Details

• PN and SDE-Net use zero-mean Gaussian noise with variance 4 as pseudo-OOD training data on MNIST and SVHN.External OOD data were also tested: SEMEION was resized to 28 × 28 for MNIST, while CIFAR10 images were grayscaled and downsampled to 28 × 28.Source: S. 2.1. Classification Setup Details · pp. 13–14
• SDE-Net replaces six residual blocks with one block, uses N = 6 subintervals, and runs one training and ten test forward propagations.σmax = 500 at test time; training values were 20 for MNIST and 5 for SVHN. DeepEnsemble used five ResNets; MC-dropout used 0.1 dropout. PGD perturbation and step sizes were 0.3 and 2/255 for MNIST, and 16/255 and 0.4/255 for SVHN.Source: S. 2.1. Classification Setup Details · p. 14
• Both datasets used SGD with momentum 0.9 and weight decay 5 × 10−4, with 200 epochs for BBP and p-SGLD and fewer epochs for other methods.Other methods used 40 epochs on MNIST and 60 on SVHN; the passage specifies method-specific learning rates and decay schedules for both datasets.Source: S. 2.1. Classification Setup Details · p. 14

• Boston Housing features are repeated six times and zero-padded to match the other dataset's feature count; training data are also perturbed with Gaussian noise of variance 4 as pseudo-OOD data.Source: S. 2.2. Regression Setup Details · p. 14
• Regression features and targets are normalized to zero mean and unit variance; SDE-Net uses four subintervals, with two layers placed before and after its block.Source: S. 2.2. Regression Setup Details · p. 14
• Training uses one forward pass, while testing uses 10 stochastic SDE-Net passes and 20 posterior samples for MC-dropout, BBP, and p-SGLD.Optimization uses momentum SGD for 60 epochs, with separate learning rates for the drift and diffusion nets.Source: S. 2.2. Regression Setup Details · p. 14

S. 2.3. Active Learning Setup

• Active learning normalizes features and targets, starts from 50 randomly selected training samples, reuses the regression OOD architecture, and trains for 100 epochs with batch size 50.Source: S. 2.3. Active Learning Setup · pp. 14–15

S. 3.1. Visulization Using Synthetic Dataset

• Synthetic-data visualizations show epistemic uncertainty low within the training region and high outside the training distribution.Source: Figure 8 · p. 15, S. 3.1. Visulization Using Synthetic… · p. 15

S. 3.2. Expected Calibration Error

• On MNIST, SDE-Net is competitive with DeepEnsemble and MC-dropout and beats other methods; on SVHN, it beats all baselines.Source: S. 3.2. Expected Calibration Error · p. 15, Figure 9 · p. 16

S. 3.3. Ablation Study

• Across MNIST OOD-training settings, SDE-Net performs well, while PN performs poorly with Gaussian-based OOD data but well with SVHN, suggesting sensitivity to the OOD data used in training.Source: S. 3.3. Ablation Study · p. 15, Table 4 · p. 16
• Without the OOD regularizer, SDE-Net's uncertainty performance deteriorates to traditional-NN levels; Brownian-motion variance directly models epistemic uncertainty for each data point.Source: S. 3.3. Ablation Study · pp. 15–16, Table 5 · p. 17

S. 4.1. Classification Task

• Classification implementation excerpts define diffusion networks for MNIST and SVHN plus a ResNet block; BBP uses an identical residual-block architecture with a factorized Gaussian weight posterior.Source: S. 4.1. Classification Task · pp. 18–19

S. 4.2. Regression Task

• Regression baselines use a six-hidden-layer fully connected network with dropout and output a mean and scale; BBP uses the same architecture with a factorized Gaussian weight posterior.Source: S. 4.2. Regression Task · pp. 19–20