Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification

Yinhao Zhu; Nicholas Zabaras

Source-linked AI summary

Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification

Yinhao Zhu, Nicholas Zabaras

arXiv:1801.06879v1Jan 21, 2018physics.comp-phcs.CVcs.LGstat.ML

TL;DR

The paper tackles surrogate modeling for stochastic PDEs with high-dimensional inputs and limited deterministic simulations. It combines a Bayesian convolutional encoder-decoder with SVGD-based approximate inference, reporting promising predictive performance and uncertainty modeling, including calibrated uncertainty estimates and successful uncertainty propagation.

Problem
High-dimensional stochastic PDE surrogates must support uncertainty quantification despite limited deterministic simulations, while conventional Monte Carlo and many surrogate methods scale poorly.
Method
A Bayesian fully convolutional DenseNet-based encoder-decoder maps high-dimensional input fields directly to output fields, using SVGD for approximate inference over millions of uncertain parameters.
Results
The approach achieves promising predictive performance and uncertainty modeling with limited training data, while Bayesian inference provides well-calibrated predictive uncertainty and supports uncertainty propagation.
Takeaways & Limitations
One DenseED-c16 network works across problems with different intrinsic dimensionality, and the Bayesian model accounts for epistemic uncertainty in small-data settings.
Takeaways & Limitations
The network does not use governing equations or constraints linking the three output fields.

Abstract

from arXiv · show

We are interested in the development of surrogate models for uncertainty quantification and propagation in problems governed by stochastic PDEs using a deep convolutional encoder-decoder network in a similar fashion to approaches considered in deep learning for image-to-image regression tasks. Since normal neural networks are data intensive and cannot provide predictive uncertainty, we propose a Bayesian approach to convolutional neural nets. A recently introduced variational gradient descent algorithm based on Stein's method is scaled to deep convolutional networks to perform approximate Bayesian inference on millions of uncertain network parameters. This approach achieves state of the art performance in terms of predictive accuracy and uncertainty quantification in comparison to other approaches in Bayesian neural networks as well as techniques that include Gaussian processes and ensemble methods even when the training data size is relatively small. To evaluate the performance of this approach, we consider standard uncertainty quantification benchmark problems including flow in heterogeneous media defined in terms of limited data-driven permeability realizations. The performance of the surrogate model developed is very good even though there is no underlying structure shared between the input (permeability) and output (flow/pressure) fields as is often the case in the image-to-image regression models used in computer vision problems. Studies are performed with an underlying stochastic input dimensionality up to $4,225$ where most other uncertainty quantification methods fail. Uncertainty propagation tasks are considered and the predictive output Bayesian statistics are compared to those obtained with Monte Carlo estimates.

1. Introduction

The paper addresses surrogate modeling for stochastic PDEs when deterministic simulations are scarce and stochastic inputs are high-dimensional. It proposes an end-to-end Bayesian convolutional encoder-decoder approach that learns input-output mappings while providing predictive uncertainty.

Motivation: Limited deterministic simulation runs make vanilla Monte Carlo impractical for uncertainty propagation and motivate surrogate modeling.Existing surrogates based on Gaussian processes and related methods have difficulty scaling to high-dimensional problems.
Motivation: Dimensionality-reduction pipelines typically separate input compression from regression and may be inefficient when the mapping back to physical input space is unavailable.The paper contrasts this workflow with supervised approaches that learn representations using the output regression task.
Motivation: Deep neural networks and CNNs can automatically learn multiscale representations from high-dimensional inputs under supervision, reducing reliance on handcrafted basis functions or expert features.Their expressiveness supports complex nonlinear mappings even when simulation datasets contain relatively few runs.
Proposed approach: The proposed model directly maps high-dimensional stochastic input fields to output fields with a fully convolutional encoder-decoder, avoiding explicit intermediate dimensionality reduction.DenseNet blocks are used within the encoder and decoder, while the Bayesian treatment targets uncertainty estimation under limited training data.
Proposed approach: The paper frames surrogate prediction and uncertainty propagation for stochastic PDE systems as image-to-image regression, including flow in random porous media.The approach is evaluated in experiments described later in the paper, including Bayesian predictive uncertainty.

2. Methodology

The methodology represents discretized stochastic PDE inputs and outputs as multichannel images and learns their mapping with a fully convolutional DenseED encoder-decoder. Dense connectivity, convolutional transitions, and Bayesian surrogate modeling support high-dimensional regression from limited simulation data.

Surrogate Modeling as Image-to-Image Regression: Discretized random input and response fields are represented as high-dimensional vectors over fixed spatial grids, with input and output channels at each grid location.The formulation supports multiple input channels and spatial dimensions, although the primary interest is one input property field.
Surrogate Modeling as Image-to-Image Regression: The surrogate f(x, θ) is trained on limited pairs of random-field realizations and simulated responses to approximate the simulator-induced mapping η(x).The training set contains N simulation runs, with θ denoting model parameters.
Surrogate Modeling as Image-to-Image Regression: For 2D grids, each input realization is organized as x_i ∈ R^(d_x×H×W) and each simulated output as y_i ∈ R^(d_y×H×W), treating dimensions as image channels.The same organization extends to 3D by adding a depth axis.
Surrogate Modeling as Image-to-Image Regression: Image-to-image regression predicts output fields at each grid point rather than a single imagewise label, using an encoder-decoder coarse-to-refine architecture.The encoder reduces spatial dimensions to coarse features, and the decoder reconstructs spatially resolved outputs.
Densely Connected Convolutional Networks: DenseNet connects each layer to all subsequent layers by concatenating earlier feature maps, improving information flow and making feature-map growth depend on the growth rate K.A dense block contains L same-size layers, while transition layers change feature-map size and count between blocks.
Densely Connected Convolutional Networks: Encoding layers downsample and decoding layers upsample feature maps through stride-2 convolutions, with no pooling so spatial location information is retained.The first transition convolution reduces feature-map count without changing size; the second changes size without changing count.
Network architecture: DenseED: DenseED combines fully convolutional encoder-decoder paths with adapted DenseNet blocks and omits feature-map concatenation between encoder and decoder paths.The architecture was selected after hyperparameter and architecture search, with no overfitting observed in the reported calculations.

3. Bayesian Neural Networks

The paper formulates neural-network parameters probabilistically to represent predictive uncertainty and uses particle-based Stein variational inference for Bayesian deep networks.

Bayesian formulation: Bayesian neural networks treat weights and biases as random variables to model epistemic uncertainty from limited training data.Additive noise is also introduced to model aleatoric uncertainty.
Bayesian formulation: The model uses Gaussian parameter priors with Gamma-distributed precision, producing a heavy-tailed Student’s t prior with greater mass near zero.The prior is specified as p(w | α) = N(w | 0, α^-1I) and p(α) = Gamma(α | a0, b0).
Additive noise model: Additive noise is modeled with output-wise, channel-wise, or input-dependent pixel-wise structures using Gaussian or Laplacian perturbations.The pixel-wise formulation yields heteroscedastic noise, and the variance is constrained positive with a softplus transformation.
Stein variational inference: SVGD approximates the posterior with particles transformed along a Stein-derived direction that combines kernel-weighted posterior gradients with a repulsive kernel term.The repulsive term maintains particle diversity, while one particle reduces the method to a MAP estimate.
Stein variational inference: The SVGD procedure is implemented with GPU acceleration and scaled to DenseED networks using 20 deterministic neural-network particles.Each iteration computes posterior gradients, kernel interactions, and local parameter updates until convergence.
Uncertainty quantification: The resulting Bayesian surrogate supports posterior predictive distributions and predictive moments, including the mean and variance at a specified input.These quantities are used for uncertainty quantification and propagation.

4. Numerical Implementation and Results

The study evaluates Darcy-flow surrogate modeling across permeability datasets with intrinsic dimensionalities from 50 to 4,225. DenseED-c16 performs well across these settings, while the Bayesian surrogate additionally estimates predictive uncertainty and reproduces Monte Carlo uncertainty-propagation statistics.

4.1. Datasets: The Darcy-flow experiments use a Gaussian random log-permeability field on a unit square, with pressure and velocity fields computed by a mixed finite-element simulator.The simulator uses third-order Raviart–Thomas elements for velocity and fourth-order discontinuous elements for pressure.
4.1. Datasets: The datasets vary intrinsic input dimensionality across KLE50, KLE500, and KLE4225, with KLE4225 retaining all 4,225 grid-point degrees of freedom.The permeability field exhibits high pixel-to-pixel variability, making the highest-dimensional case challenging for data-driven models.
4.3. Non-Bayesian Surrogate Model: DenseED-c16 achieves validation R2-scores above 0.9 with reasonably small training sets for dimensionalities ranging from 50 to 4,225.The score improves with more training data at fixed dimensionality and is closer to 1 for smaller intrinsic dimensionality.
4.3. Non-Bayesian Surrogate Model: DenseED-c16 predicts rapidly varying pressure and velocity fields well even for KLE4225 inputs.The same network architecture is used across the three intrinsic input dimensionalities.
4.4. Bayesian Surrogate Model: The Bayesian surrogate improves R2-score over the non-Bayesian version while providing predictive variance unavailable from deterministic models.Predictive accuracy increases and predictive uncertainty decreases as the training dataset grows.
4.4. Bayesian Surrogate Model: For KLE4225 uncertainty propagation, predictive density estimates are close to Monte Carlo results and become closer with more training data.Velocity predictions are better than pressure predictions at the reported locations, while reliability diagrams indicate generally good calibration.

5. Conclusions

The paper presents Bayesian convolutional encoder-decoder surrogates for high-dimensional stochastic-PDE systems, combining predictive performance with uncertainty estimates under limited training data. Experiments show broad performance across intrinsic dimensionalities, while identifying missing physics information and difficult Bayesian inference as important boundaries.

The end-to-end Bayesian image-to-image surrogate avoids the usual linear dimensionality reduction step while targeting predictive performance and uncertainty modeling with limited training data.
One DenseED-c16 network works well across problems with different intrinsic dimensionality.
The Bayesian model provides epistemic uncertainty estimates for predictions trained on small datasets, and SVGD-based inference supports Bayesian neural-network training.
Reliability diagrams show that the trained model's uncertainty is well-calibrated.
The current network does not use governing equations or constraints between the three output fields, while Bayesian inference still requires a posterior over millions of variables from hundreds of training data.

Appendix A. Experiments on Darcy Flow Problem

Appendix experiments evaluate DenseED architectures, training settings, and feature-map scales for Darcy-flow surrogate modeling across datasets with varying intrinsic dimensionality. DenseED-c16 performs well across all three datasets, whereas DenseED-c8 fails on the highest-dimensional KLE4225 case because its code dimension is too small.

Architecture and training: The encoder downsamples feature maps by 2, while the decoder upsamples by 2 with transposed convolution to preserve location information for regression.
Architecture and training: Coarsest-scale feature-map dimensions are chosen through down-sampling depth to provide suitable receptive fields for fields with different spatial correlations.KLE50 fields are smoother, whereas KLE500 and KLE4225 vary more rapidly while retaining weak long-range correlation.
Architecture and training: The experiments use no skip connections or dropout, with specified convolution kernels, growth rate 16, regularized MSE loss, R2-score validation, Adam optimization, and 200 training epochs.Batch size is smaller than the training-set size and weight decay is 0.0005.
Architecture comparison: DenseED-c16 works well for all three datasets, while DenseED-c8 performs poorly on KLE4225 because its code dimension is too small and produces overly smooth output fields.The compared networks are DenseED-c16, DenseED-c8, and DenseED-c32.
Architecture search: The architecture search constrains candidate networks to fewer than 0.25 million parameters, and Hyperband identifies only sub-optimal hyperparameters indicating a potential range.The reported Hyperband search uses a one-third retention rule for the best configurations.
Architecture search: The hyperparameter search reports an approximate initial learning rate of 0.002, weight decay 0.0005, batch size 16, and Adam optimizer.The paper notes that these searched hyperparameters are sub-optimal and gives the actual training settings separately.

Appendix B. Generalization behavior

DenseED-c16 remains free of observed overfitting as its parameter count increases, supporting a 241,164-parameter baseline configuration.

37,892 to 805,204 parameters produce no observed overfitting: training error does not decrease while test error increases.This behavior was measured with 256 training data points using training and test RMSE.
Below 200,000 parameters, both training and test losses still have substantial room for improvement.
The baseline DenseED-c16 configuration uses 241,164 parameters because this lies in a favorable region of the generalization curve.