Uncertainty propagation in auto-regressive random neural network models

Janice Adams; Daniele Venturi

Source-linked AI summary

Uncertainty propagation in auto-regressive random neural network models

Janice Adams, Daniele Venturi

arXiv:2608.20483v1Aug 20, 2026stat.MLcs.LGcs.NEphysics.comp-ph

TL;DR

The paper addresses uncertainty propagation when both neural-network inputs and parameters are random, particularly for autoregressive dynamical systems. It combines Leaky ReLU linearization, analytical distribution and moment calculations, and particle resampling, demonstrating accurate propagation through the predictability horizon while identifying limitations of single-Gaussian propagation.

Problem
Reliable uncertainty quantification is needed for neural-network predictions in autoregressive nonlinear dynamical systems, where input and parameter uncertainties may be amplified.
Method
The framework linearizes Leaky ReLU networks jointly in inputs and parameters, derives analytical output distributions and moments, and recursively propagates state, parameter, and cross-covariance uncertainty with particle resampling.
Results
The method accurately reproduces Monte Carlo uncertainty estimates for Lorenz-63 and demonstrates applicability to the high-dimensional Kuramoto–Sivashinsky system.
Takeaways & Limitations
State–parameter cross-covariance is essential for multi-step uncertainty propagation, while periodic resampling mitigates variance growth from direct single-Gaussian propagation.
Takeaways & Limitations
Single-Gaussian propagation captures only the first two moments, adds approximation error for non-Gaussian initial distributions, and can become unstable over long horizons.

Abstract

from arXiv · show

We develop analytical and particle-based methods for uncertainty propagation in random neural network models, where both the inputs and network parameters are allowed to be random. Building on the piecewise-linear structure of the Leaky ReLU activation function, we derive a local approximation of the neural network output with respect to perturbations in both its inputs and parameters. This approximation is exact for perturbations that preserve the network activation pattern, and it allows us to compute analytical expressions for the probability density function and characteristic function of the network output, together with closed-form approximations for its mean and covariance. We extend this uncertainty propagation framework to autonomous dynamical systems whose one-step evolution map is represented by a random neural network. Repeated application of this map defines an autoregressive model, for which we derive recursive equations to propagate uncertainty in both the state and network parameters over time. These equations explicitly account for the state-parameter cross-covariance that develops under successive iterations of the network. Numerical experiments on the Lorenz-63 system and the Kuramoto-Sivashinsky equation demonstrate accurate uncertainty propagation through the predictability horizon and the applicability of the proposed framework to high-dimensional dynamical systems.

1. Introduction

The paper addresses uncertainty quantification for neural-network predictions, especially in nonlinear autoregressive dynamical systems where input and parameter uncertainties can amplify. It develops analytical and particle-based propagation methods using Leaky ReLU's piecewise-linear structure.

Uncertainty assessment for neural-network predictions has lagged behind their expanding use across scientific and engineering applications.
Small uncertainties in inputs or network parameters can be amplified in autoregressive neural-network models of nonlinear dynamical systems, producing inaccurate or unstable predictions.
Leaky ReLU's piecewise linearity makes first-order output linearization exact for perturbations that preserve the network activation pattern.
The framework derives output PDFs, characteristic functions, and closed-form mean and covariance approximations using a combined input–parameter Jacobian.
The paper extends uncertainty propagation to autoregressive settings and describes the paper's organization around network modeling, linearization, output distributions, moments, and repeated application.

2. Neural Network Model

The model is a fully connected feed-forward neural network mapping an M-dimensional input to an M-dimensional output through H hidden layers of dimension N, with affine maps and Leaky ReLU activations.

The network maps input x to output y through a fully connected feed-forward architecture with H hidden layers.
The input and output vectors have dimension M, while each hidden layer has dimension N.
Each hidden layer applies an affine transformation followed by a nonlinear activation, with r_i = ϕ(W_i r_i−1 + b_i).
The output layer is affine, given by y = W_H+1 r_H + b_H+1.
The parameters comprise the weights and biases of all H+1 layers, with dimensions specified for the input, hidden, and output connections.
All layers use Leaky ReLU activations, with the slope parameter α defaulting to 0.01.

3. Linearization of the Neural Network Output

The paper linearizes network output jointly with respect to input and parameter perturbations. For Leaky ReLU networks, this approximation is exact within activation regions.

Input-output linearization is used in neural networks for interpretability, physics-informed structure, and inference, while parameter expansions support optimization and uncertainty quantification.
Leaky ReLU's piecewise linearity makes local linearization exact when perturbations do not change activation branches.
The paper treats the network as a function of both input x and parameters θ, introducing perturbations Δx and Δθ.
The first-order approximation combines input and parameter effects as F_{θ+Δθ}(x+Δx) ≈ F_θ(x) + J_xΔx + J_θΔθ.
J_x and J_θ are the Jacobians of the network with respect to the input and parameters, respectively.

4. PDF of the Neural Network Output

The paper derives the output PDF by marginalizing the deterministic network map over the joint input–parameter distribution, then uses characteristic functions to represent the linearized output distribution.

The output-PDF analysis first treats x and θ as random vectors and y as the resulting random network output.
Marginalizing the conditional output distribution over the joint PDF of x and θ yields the exact output PDF representation.
After decomposing inputs and parameters into means and perturbations, the linearized output PDF is expressed through the perturbation distribution and the network Jacobians.
The inner perturbation integral is the joint characteristic function evaluated at a Jacobian-dependent argument, producing an analytical approximate output PDF and characteristic function.
The framework specializes the PDF approximation to jointly Gaussian, independent general, and independent uniform input–parameter perturbations.
For independent uniform perturbations, the characteristic function is constructed from centered-uniform factors using the sinc function.
The formulation is related to Bayesian neural networks but allows uncertainty in both random inputs and parameters, whereas the standard posterior predictive case treats the input as deterministic.

5. Statistical Moments

The linearized model yields closed-form approximations for neural-network output moments, incorporating uncertainty from inputs, parameters, and their cross-covariance.

Closed-form approximations are derived for the mean and covariance of the neural-network output under the linearized model.
The moment equations remain tractable in the auto-regressive setting even when the joint perturbation density lacks a closed-form expression.
The first-order approximation assumes zero-mean input and parameter perturbations.
The output covariance combines input uncertainty, parameter uncertainty, and cross-covariance terms between input and parameter perturbations.

6. Auto-Regressive Neural Networks

The paper extends uncertainty propagation to repeatedly applied neural-network dynamics, where state–parameter dependence prevents simple state-only propagation. It develops recursive moment updates and particle-based alternatives for representing evolving, potentially non-Gaussian state distributions.

Auto-Regressive Neural Networks: Auto-regressive neural networks map each current state to the next state and recursively generate time-series trajectories.
Auto-Regressive Neural Networks: The framework propagates the state PDF from an initial joint distribution of state and network parameters.
Auto-Regressive Neural Networks: State–parameter correlation prevents factorizing the joint density or propagating the state PDF using only a state transition density.
6.1. Statistical Moments: Recursive linearized updates propagate the state mean, covariance, and state–parameter cross-covariance over multiple time steps.
6.2. Propagation Methods: Single-Gaussian propagation can introduce approximation error for non-Gaussian initial distributions and produce rapidly growing variances with unstable predictions.
6.2. Propagation Methods: The particle method evolves local Gaussian moments, resamples particles, resets their covariances, and retains non-Gaussian features while limiting covariance growth.

7. Numerical Experiments

The experiments evaluate uncertainty propagation for random neural-network models on Lorenz-63 and Kuramoto–Sivashinsky systems. Resampled Moment Propagation generally tracks Monte Carlo uncertainty, while computational costs and resampling choices constrain high-dimensional use.

Experimental setup: The experiments study autoregressive uncertainty propagation for Lorenz-63 and Kuramoto–Sivashinsky systems with random initial states and network parameters.Initial state and parameter perturbations are assumed independent initially, but become statistically dependent after one propagation step.
Lorenz-63 System: Resampling interval is the primary performance parameter for Lorenz-63, while resampling too frequently can cause particle impoverishment and underestimate variance.Intervals of approximately 10–50 steps produced comparable results with as few as 100 particles.
Lorenz-63 System: The naive single-Gaussian method initially overestimates Lorenz-63 variance and becomes unstable around t ≈ 2, with variance far exceeding the Monte Carlo spread.Small Gaussian mixtures without resampling do not significantly improve the results.
Lorenz-63 System: For Lorenz-63, Resampled Moment Propagation closely reproduces Monte Carlo means, covariances, and overall distributions through the predictability horizon Tp ≈ 5.11.The comparison uses 1000 particles for the propagation method and 3000 samples for the Monte Carlo baseline.
Kuramoto–Sivashinsky System: The main high-dimensional limitation is memory: storing state–parameter cross-covariances requires approximately 48 GB in double precision for 100 particles and a 200-dimensional state.Neglecting cross-covariance or resampling every step substantially underestimates propagated variance.

8. Conclusion

The paper develops uncertainty propagation for random neural networks and extends it to autoregressive dynamical systems with uncertain states and parameters. Resampled Moment Propagation represents evolving non-Gaussian distributions while addressing covariance growth, but further work is needed for scalability, principled resampling, and measurement updates.

Framework: The framework propagates uncertainty in random MLP inputs and parameters using a Leaky ReLU-based local approximation and applies it to autoregressive dynamical systems.The approximation yields analytical output distributions and closed-form moment approximations, while repeated network applications require state–parameter cross-covariance propagation.
Framework: State–parameter cross-covariance develops under repeated network applications and is explicitly included to capture accumulated parameter uncertainty.
Resampled Moment Propagation: Resampled Moment Propagation combines local analytical moment propagation with periodic conversion of local Gaussians into a particle cloud, resetting local covariances after resampling.This represents evolving non-Gaussian state distributions while limiting the covariance growth observed with single-Gaussian propagation.
Future directions: Analytical covariance-growth bounds could replace empirical resampling-interval tuning, while reduced-order or low-rank cross-covariance representations could address memory scaling with state dimension and parameter count.
Future directions: The framework could be extended to other activation functions, higher-order expansions, exact per-layer moment matching, and explicit measurement updates for recursive Bayesian state estimation.

Appendix A. Particle Degeneracy under Resampling

The appendix analyzes lineage loss in GMM resampling by computing the probability that a particle contributes no descendants. In the large-particle and large-sample limit, approximately 37% of lineages disappear at each resampling event, creating a particle-degeneration trade-off.

Lineage loss: GMM resampling forms an SL-sample pool from S particles and retains S samples uniformly without replacement, so some particle lineages may have no descendants.For a fixed particle, Xi counts retained descendants, and the appendix evaluates Pr(Xi = 0).
Derivation: The no-descendant probability is derived by partitioning the pool into one particle’s L local samples and the remaining L(S − 1) samples.
Limits: For fixed S, taking L → ∞ yields the limiting no-descendant probability, followed subsequently by the large-S limit.
Implication: Approximately 37% of existing particle lineages are expected to disappear at each resampling event in the large-S, large-L limit.Repeated resampling can therefore cause substantial particle degeneration, consistent with sample impoverishment observed for Lorenz-63.