Source-linked AI summary
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, Paris Perdikaris
TL;DR
High-dimensional PDE surrogate modeling is difficult because supervised training requires scarce simulation outputs and standard models may not enforce physical constraints. The paper incorporates governing equations into neural loss and likelihood functions, training deterministic and probabilistic surrogates from input data alone. The resulting models achieve comparable predictive performance, better out-of-distribution predictions, and uncertainty characterization, while remaining limited in expressing uncertainty for unfamiliar inputs.
Problem
High-dimensional PDE surrogate modeling requires scarce labeled simulation outputs and must address physical constraints, uncertainty quantification, and generalization.
Method
The paper trains convolutional encoder-decoder and conditional flow-based models without labeled outputs by incorporating PDE constraints into loss or likelihood functions and minimizing reverse KL divergence.
Results
The physics-constrained surrogates achieve high predictive accuracy, better out-of-distribution generalization than data-driven surrogates, and calibrated predictive uncertainty across uncertainty propagation and calibration tasks.
Takeaways & Limitations
PDE-constrained neural surrogates can provide accurate deterministic predictions and predictive distributions without solving the governing PDEs for training inputs.
Takeaways & Limitations
Under out-of-distribution inputs, the current surrogate does not produce larger uncertainty even though predictive-mean errors can be an order of magnitude larger than uncertainty.
Abstract
from arXiv · showhide
Surrogate modeling and uncertainty quantification tasks for PDE systems are most often considered as supervised learning problems where input and output data pairs are used for training. The construction of such emulators is by definition a small data problem which poses challenges to deep learning approaches that have been developed to operate in the big data regime. Even in cases where such models have been shown to have good predictive capability in high dimensions, they fail to address constraints in the data implied by the PDE model. This paper provides a methodology that incorporates the governing equations of the physical model in the loss/likelihood functions. The resulting physics-constrained, deep learning models are trained without any labeled data (e.g. employing only input data) and provide comparable predictive responses with data-driven models while obeying the constraints of the problem at hand. This work employs a convolutional encoder-decoder neural network approach as well as a conditional flow-based generative model for the solution of PDEs, surrogate model construction, and uncertainty quantification tasks. The methodology is posed as a minimization problem of the reverse Kullback-Leibler (KL) divergence between the model predictive density and the reference conditional density, where the later is defined as the Boltzmann-Gibbs distribution at a given inverse temperature with the underlying potential relating to the PDE system of interest. The generalization capability of these models to out-of-distribution input is considered. Quantification and interpretation of the predictive uncertainty is provided for a number of problems.
1. Introduction
The paper targets data-efficiency, uncertainty quantification, and generalization challenges in high-dimensional surrogate modeling by incorporating PDE constraints into label-free neural surrogates. It combines convolutional encoder-decoder and conditional flow-based approaches, reporting comparable prediction and improved out-of-distribution behavior.
- High-dimensional surrogate modeling faces challenges in data-efficiency, uncertainty quantification, and generalization, particularly with stochastic spatial inputs.
- Supervised surrogate training requires simulation outputs, while Gaussian processes and deep neural networks still face difficulties modeling high-dimensional input-output maps.
- Physics knowledge can be incorporated through architectures or constraint learning that penalizes violations of physical constraints and symmetries.
- Deeper networks and stochastic optimization have enabled neural approaches to address several high-dimensional PDEs with improved accuracy and speed.
- The proposed physics-constrained surrogates use CNN-based parameterizations and conditional generative modeling without simulation outputs, with comparable prediction and better extrapolative performance.
2. Problem Definition
The paper formulates PDE solution, surrogate modeling, and uncertainty propagation for random high-dimensional input fields. It distinguishes deterministic and probabilistic surrogates while emphasizing the absence of labeled solution data and the role of physical energy and boundary constraints.
- The PDE system uses field variables, source fields, constitutive input properties, differential operators, and boundary operators over a spatial domain.
- Potential-based PDEs can be solved by minimizing a field energy functional, while residual norms can provide the potential when no energy functional is available.
- The deterministic surrogate predicts discretized PDE responses y from random-vector inputs x using only training inputs sampled from p(x).
- The probabilistic surrogate specifies a conditional density pθ(y|x) to represent possible PDE solutions rather than only a point estimate.
- The learning task is between unsupervised and supervised learning because input-output labels are absent while the input-to-output map must still be discovered.
- Forward uncertainty propagation estimates response moments or other aspects of the output density induced by the input distribution.
3. Methodology
The methodology parameterizes PDE solutions and high-dimensional surrogates with neural networks, then trains them using physics-based losses or a reverse-KL objective rather than labeled outputs. A conditional flow model additionally represents predictive uncertainty while enforcing PDE and boundary constraints.
- 3.1. Differentiable Parameterizations of Solutions: FC-NNs represent solutions at queried coordinates, whereas CNN decoders generate grid-based fields and capture spatial correlations through convolutional operations.FC-NNs support exact spatial derivatives through automatic differentiation; CNN gradients can be approximated efficiently with fixed Sobel filters and provide multiresolution representations.
- 3.2. Physics-constrained Learning of Deterministic Surrogates without Labeled Data: The surrogate extends an encoder-decoder network to map image-like stochastic inputs x directly to predicted solution fields y.The encoder transforms x into a latent variable z, which the decoder maps to y.
- 3.2. Physics-constrained Learning of Deterministic Surrogates without Labeled Data: Physics-constrained surrogates are trained without output data by minimizing equation and boundary losses for sampled inputs.The equation loss may use a PDE residual norm or variational functional, while the boundary loss is softly weighted by λ; spatial differentiation and integration use discrete operations where needed.
- 3.2.1. Physics-constrained Losses: The method supports primal and mixed Darcy-flow formulations, with strong-form residual or variational losses for pressure and, when applicable, flux.The mixed formulation introduces flux τ alongside pressure and can be expressed through residual or Hellinger-Reissner variational objectives.
- 3.3. Probabilistic Surrogates with Reverse KL Formulation: Probabilistic surrogates minimize reverse KL divergence to a Boltzmann reference density whose energy penalizes PDE and boundary violations.The inverse temperature β controls the reference-density variance, while the reverse-KL objective combines expected physics loss, conditional entropy, and a constant variational-free-energy term.
- 3.3.1. Conditional Flow-based Generative Models: Conditional Glow evaluates likelihoods efficiently through affine-coupling Jacobians, enabling entropy estimation, but predictive calibration still requires an additional constraint and validation data.The model’s exact log-likelihood supports Monte Carlo entropy estimation; output data are unnecessary for training, whereas input-output validation pairs are needed to calibrate uncertainty.
4. Numerical Experiments
The numerical experiments evaluate physics-constrained CNN and conditional Glow surrogates for deterministic PDE solving, surrogate prediction, uncertainty quantification, and out-of-distribution generalization. Across these tasks, the models use input data and PDE constraints to achieve accurate predictions, with explicit trade-offs in convergence, flux accuracy, calibration, and uncertainty expression.
- 4.2. Deterministic Surrogate: Physics-constrained surrogates achieve similar pressure-field accuracy to data-driven surrogates with the same training inputs, although they converge more slowly and predict flux fields less accurately.The PCS is trained for 300 epochs versus 200 for the DDS; flux-field error is attributed mainly to boundary derivative approximations.
- 4.2. Deterministic Surrogate: Increasing PCS training inputs decreases relative L2 pressure error, and sufficient input-only training yields performance comparable to data-driven surrogates without simulation output.The DDS requires an equal number of simulated output solutions for supervision, whereas the PCS uses physical knowledge and random input samples.
- 4.2. Deterministic Surrogate: Both PCS and DDS generalize well across related GRF inputs but less well to channelized fields, with PCS generalizing better on warped GRF and channelized inputs.This pattern persists across PCS training sizes from 512 through 4096 samples.
- 4.3. Probabilistic Surrogate: The probabilistic surrogate achieves 0.0038 relative L2 pressure error versus 0.0035 for the deterministic surrogate, while reproducing output statistics and variance well.Predictive variance reflects the boundary conditions, and the estimated mean of the horizontal flux field has around 3% relative error.
5. Conclusions
The paper presents physics-constrained surrogates that require no labeled simulation data, using governing equations in loss or likelihood functions. It reports high-dimensional predictive accuracy, uncertainty capture, and stronger out-of-distribution generalization, while identifying unresolved reliability and scaling challenges.
- The framework incorporates governing equations into loss or likelihood functions, eliminating the need for labeled training data such as simulation outputs.
- Convolutional encoder-decoder surrogates achieve high predictive accuracy for high-dimensional stochastic input fields.
- Physics-constrained surrogates show consistently better generalization than data-driven surrogates on out-of-distribution inputs.
- The flow-based conditional generative surrogate captures predictive uncertainty in uncertainty-propagation and calibration tasks.
- Open problems include improving out-of-distribution uncertainty, scaling flow-based models to higher dimensions, and extending the framework to dynamical systems.
Appendix A. Sobel filter to estimate spatial gradients
The method uses Sobel convolutions to estimate spatial gradients efficiently within CNN representations, then applies boundary corrections to improve accuracy. The correction reduces learned-solution error substantially but leaves corner errors.
- Gradient estimation: Sobel filters estimate horizontal and vertical spatial gradients using 3 × 3 convolution kernels.The convolutional operation matches CNN representations of solution fields.
- Gradient estimation: Sobel-based gradients are more efficient than automatic differentiation, but less accurate, especially near boundaries.This is the stated trade-off relative to FC-NN parameterizations.
- Boundary correction: Boundary correction estimates the horizontal gradient as (I ⋆ H)M_H with replicate padding, corresponding to forward and backward finite differences at opposite boundaries.The vertical gradient is corrected analogously.
- Boundary correction: The boundary correction reduces learned-solution error by several times, although errors remain at the four corners.More refined correction is identified as a possible improvement.
Appendix B.1. Network architecture
The appendix specifies fully connected and convolutional architectures used for deterministic PDE experiments. The CNN decoder maps a compact latent representation to solution fields and is trained with mixed residual loss.
- FC-NN: The experimental FC-NN has 8 hidden layers, 512 nodes per layer, Tanh activation, and 1,841,155 parameters.Its input and output dimensions are 2 and 3, respectively.
- Training: Both FC-NNs and CNNs are trained with mixed residual loss using L-BFGS, learning rate 0.5, and boundary-loss weight λ = 10.The optimizer uses history size 50 and maximum iteration 20.
Appendix B.2. Supplementary Numerical Experiments
For GRF KLE4096, the CNN captures the flux field faster and better than the FC-NN, but produces severe checkerboard artifacts in the pressure field.
- GRF KLE4096: The CNN captures the GRF KLE4096 flux field faster and better than the FC-NN, while its pressure field shows severe checkerboard artifacts.The largest pressure error for the CNN exceeds that of the FC-NN pressure solution.
Appendix C.1. Details on the network structure
The network structure combines dense blocks, spatial downsampling, feature squeezing, latent-variable splitting, and invertible flow steps. The appendix also labels CNN results at iterations 50 and 500 and references a Darcy-flow figure.
- Encoder: The encoder uses L dense blocks that preserve feature-map size and L − 1 Trans Down layers that typically halve it.Features extracted after each dense block are treated as inputs ξ_l.
- Flow structure: Each flow step combines ActNorm, an invertible 1 × 1 convolution, and an affine coupling layer.The coupling network produces scale and translation channels after three dense layers and a 3 × 3 convolution.
- Supplementary experiments: The supplementary materials label CNN outputs at iterations 50 and 500 and include a Darcy-flow example for GRF KLE4096.These labels identify experiment stages rather than reporting a numerical comparison.
Appendix C.2. Results on higher input dimension
Appendix C.2 reports results for a conditional Glow trained on higher-dimensional inputs, using GRF KLE256 samples on a 32 × 32 grid and testing two inputs.
- Appendix C.2. Results on higher input dimension: 4096 samples from GRF KLE256 were used to train the conditional Glow.
- Appendix C.2. Results on higher input dimension: The GRF KLE256 inputs were represented on a 32 × 32 grid.
- Appendix C.2. Results on higher input dimension: Prediction results for two test inputs are shown in Fig. C.18.