PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Reduction and Neural Networks for Parametric PDEs

Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, Andrew M. Stuart

arXiv:2005.03180v2math.NAcs.LGstat.ML

TL;DR

The paper addresses repeated evaluation of expensive input-output maps, such as parametric PDE solution operators, when inputs and outputs are infinite-dimensional. It combines function-space model reduction with a neural network between latent representations, and proves convergence for a class of maps while demonstrating mesh-independent numerical behavior. The experiments include elliptic PDE coefficient-to-solution maps and viscous Burgers' equation.

  • Problem

    Repeatedly evaluating expensive forward models for statistically similar inputs requires data-driven emulators that can operate on infinite-dimensional input and output spaces.

  • Method

    The method approximates encoder-decoder maps for input and output function spaces and learns the latent-code map with a neural network.

  • Results

    The paper proves convergence for a class of input-output maps and demonstrates mesh-independent approximation in numerical PDE experiments.

  • Takeaways & Limitations

    The approach provides a data-driven PDE emulator whose approximation remains stable as the finite-dimensional discretization is refined.

  • Takeaways & Limitations

    The theory assumes globally Lipschitz maps in its main statement, while the neural-network approximation theorem does not address whether training optimization finds the required parameters.

Abstract

from arXiv · show

We develop a general framework for data-driven approximation of input-output maps between infinite-dimensional spaces. The proposed approach is motivated by the recent successes of neural networks and deep learning, in combination with ideas from model reduction. This combination results in a neural network approximation which, in principle, is defined on infinite-dimensional spaces and, in practice, is robust to the dimension of finite-dimensional approximations of these spaces required for computation. For a class of input-output maps, and suitably chosen probability measures on the inputs, we prove convergence of the proposed approximation methodology. We also include numerical experiments which demonstrate the effectiveness of the method, showing convergence and robustness of the approximation scheme with respect to the size of the discretization, and compare it with existing algorithms from the literature; our examples include the mapping from coefficient to solution in a divergence form elliptic partial differential equation (PDE) problem, and the solution operator for viscous Burgers' equation.

1. Introduction

The paper proposes a data-driven emulator for input-output maps between infinite-dimensional function spaces, combining model reduction with neural networks to achieve mesh-independent approximation. It motivates this framework through parametric and time-dependent PDEs and positions it against mesh-dependent neural-network and classical numerical approaches.

  • Proposed approach: The emulator is designed to approximate PDE solution maps between function spaces while remaining robust to the resolution used for finite-dimensional computation.The paper describes this as a mesh-independent approximation property.
  • Proposed approach: The method reduces input and output dimensions, then learns a neural-network map between the resulting finite-dimensional latent spaces.It combines dimension reduction and reconstruction techniques with neural-network approximation.
  • Literature context: Existing image-to-image regression discretizes both spaces into Euclidean vectors, so its approximation rates and network architecture depend on the mesh.The paper identifies this dependence as a limitation for consistent approximation across discretizations.
  • Literature context: The paper situates its approach alongside solution parametrization by neural networks, function-space neural operators, reduced-basis methods, and latent ODE learning.These lines of work address PDE approximation using different combinations of neural networks, basis functions, and reduced dynamics.
  • Contributions: The paper's primary contributions include learning Hilbert-space maps, controlling approximation errors, proving arbitrary-accuracy approximation in function spaces, and demonstrating mesh independence numerically.The numerical experiments compare the proposed methodology with other parametric-PDE methods.
  • Contributions: The analysis uses PCA to finite-dimensionalize inputs and outputs, proves a result under a global Lipschitz assumption, and includes experiments beyond that theory's scope.An appendix extends the analysis to locally Lipschitz maps.

2. Proposed Method

The method approximates infinite-dimensional input-output maps by reducing input and output dimensions, then learning a latent-space map from data. PCA supplies analyzable encoders and decoders, while a neural network approximates the latent relationship for non-intrusive black-box use.

  • 2.1. Overview: Training samples are i.i.d. under an input measure µ, so accuracy is designed with respect to norms induced by µ and Ψ♯µ.The output samples follow the push-forward measure Ψ♯µ.
  • 2.1. Overview: The framework approximates maps Ψ:X→Y using finite-dimensional encoders, decoders, and a learned map between latent codes.It exploits approximate finite-dimensional structure by reducing both input and output spaces before interpolation.
  • 2.1. Overview: PCA is used to encode and decode function-space inputs and outputs, with a neural network replacing the generally unknown latent map.The practical method combines PCA encoding/decoding with a data-driven neural-network approximation χ≈ϕ.
  • 2.1. Overview: PCA’s linear approximation of the solution manifold may converge slowly, motivating nonlinear dimension-reduction methods such as function-space deep autoencoders as future work.The paper focuses on PCA because it has an unambiguous decoder and supports analysis, despite this scope limitation.
  • 2.2. PCA On Function Space: PCA selects empirical subspaces that minimize projection error, while approximation quality depends on covariance-spectrum decay and empirical eigenstructure.The empirical PCA subspace Vd,N solves the d-dimensional projection minimization problem.
  • 2.4. Comparison to Existing Methods: The method is non-intrusive: it uses the available input-output dataset without requiring explicit knowledge of the underlying PDE mapping.Unlike reduced-basis methods, it regresses latent representations rather than obtaining coefficients through an intrusive PDE approximation.

3. Approximation Theory

The paper proves approximation guarantees for a PCA-based neural-network emulator of infinite-dimensional input-output maps under moment and regularity assumptions. The error is decomposed into neural-network, input-projection, output-projection, and finite-data contributions.

  • Approximation theorem: For any ϵ > 0, suitable PCA dimensions, dataset size, and neural-network parameters yield an ϵ-approximation of a µ-measurable globally Lipschitz map Ψ.The result assumes real separable Hilbert spaces and a finite fourth moment for the input measure.
  • Assumptions and scope: The theorem is stated for globally Lipschitz maps, while a locally Lipschitz extension is available under a more stringent moment condition.The locally Lipschitz result is stated separately in Theorem A.1.
  • Approximation theorem: The neural network has explicit depth and active-weight bounds for fixed latent dimensions, but the total parameter complexity for error ϵ remains unquantified as dimensions grow.The unspecified dependence of constants and support size on dX and dY prevents an explicit end-to-end complexity estimate.
  • Approximation methodology: The approximation method uses PCA encoders and decoders for the input and output spaces, with a neural network approximating the map between their latent codes.The proof controls the errors from GX ◦ FX ≈ IX, GY ◦ FY ≈ IY, and χ ≈ ϕ.
  • PCA and finite-data approximation: The PCA projection analysis bounds the finite-data projection error by the optimal projection error plus a sampling term controlled in expectation.The expectation-based bound applies under broader data distributions, including Gaussian measures.
  • Error decomposition: The full approximation error combines neural-network approximation with finite-dimensional input and output projection errors.The prescribed order is to select latent dimensions, increase N to control covariance estimation, then choose δ and τ to control domain restriction and latent-map approximation.

4. Numerical Results

Numerical experiments across elliptic PDE, Darcy flow, and Burgers’ equation test the method’s effectiveness, mesh robustness, and dependence on reduced dimension and training data. The approach maintains stable errors under mesh refinement, while nonlinear problems favor neural networks and larger reduced dimensions can require more data.

  • Experimental setup: The experiments cover elliptic PDE, Darcy flow with log-normal and piecewise constant coefficients, and viscous Burgers’ equation.The study uses probability measures on the input spaces and generally draws N = 1024 training examples plus 5000 test examples.
  • Mesh refinement: The discretized method’s error remains essentially unchanged as mesh resolution increases for the linear elliptic, Poisson, Darcy, and Burgers’ problems.This behavior reflects constructing the neural-network architecture as a map between Hilbert spaces before discretization.
  • Linear problems: For linear elliptic and Poisson problems, the linear approximation consistently outperforms the neural network because the underlying solution map is linear.The competing image-to-image regression approach instead becomes increasingly inaccurate as resolution grows.
  • Nonlinear Darcy flow: For nonlinear Darcy flow, neural networks outperform linear maps, but increasing reduced dimension does not necessarily improve error because optimization becomes more complex.Additional training data flattens the error curve, consistent with data requirements increasing alongside reduced dimensions.
  • Mesh transfer: Transferring Darcy solutions trained on a 61×61 grid to a 421×421 grid increases relative error by 3% and 1% for the two Darcy problems, respectively.Training on a large mesh produces almost no error increase when transferring to smaller meshes, without retraining the neural network.
  • Burgers’ equation: For Burgers’ equation, the neural network performs substantially better than the linear map, consistent with the solution operator being more strongly nonlinear.As in Darcy flow, higher reduced dimension does not necessarily lower error, and additional data can mitigate this effect.

5. Conclusion

The paper presents a data-driven methodology for learning mappings between separable Hilbert spaces, proves consistency under globally Lipschitz forward maps, and demonstrates mesh-independent PDE performance, including beyond the theory’s scope.

  • The methodology learns mappings between separable Hilbert spaces and is proved consistent for globally Lipschitz forward maps.
  • Numerical experiments demonstrate mesh-independent behavior on parametric PDE problems, with good performance beyond the theory’s scope.
  • PCA restricts the method to Hilbert spaces and may not provide optimal dimension reduction even within that setting.
  • Future work includes analyzing how reduced dimensions, data requirements, network parameters, and optimization determine computational complexity.

Appendix A. Neural Networks And Approximation (Locally Lipschitz Case)

The appendix extends the approximation theory to measurable, locally Lipschitz solution maps and establishes neural-network approximation bounds under moment and regularity assumptions.

  • The analysis considers µ-measurable solution maps Ψ:X→Y that are locally Lipschitz through a growth function L.
  • The pushforward measure Ψ♯µ has bounded fourth moments when E_x∼µ(L(x,0)∥x∥_X)^4 is finite.
  • Theorem A.1 provides a neural-network approximation with an error bound whose constant is independent of d_X, d_Y, N, δ, and ϵ.
  • The proof combines the projection error estimate with neural-network approximation and bounds the resulting terms using Lipschitz estimates and moment conditions.
  • A finite-dimensional approximation is formed by reducing input and output spaces, then approximating the reduced map ϕ by a neural network χ.
  • The construction controls truncation and neural-network errors by restricting inputs to a high-probability bounded set and decomposing the expectation over that set and its complement.

Appendix B. Supporting Lemmas

The appendix supplies supporting Hilbert-space results, including covariance approximation, PCA projection properties, and Lipschitz estimates for the reduction and reconstruction maps.

  • Fan’s theorem characterizes the optimal d-dimensional orthonormal subspace through the leading eigenvalues and eigenfunctions of a compact covariance operator.
  • A Monte Carlo estimate controls the Hilbert-Schmidt distance between the population covariance operator C and its empirical counterpart C_N.
  • The maps F_X, F_Y, G_X, and G_Y are globally Lipschitz, while the reduced solution map ϕ is locally Lipschitz under the stated growth assumptions.
  • The probability of inputs exceeding the reduced-coordinate bound M is controlled using second moments and Chebyshev’s inequality.

Appendix C. Analyticity of the Poisson Solution Operator

The appendix analyzes a Poisson solution operator with coefficient-dependent forcing and establishes well-posedness, regularity, and truncation behavior under summability assumptions.

  • The Poisson operator maps coefficient sequences ξ=(ξ_1,ξ_2,...) to the unique weak solution u of a Dirichlet problem.
  • The assumption (∥φ_j∥_L∞)_{j≥1}∈ℓ_p for some p∈(0,1) ensures the coefficient expansion is sufficiently controlled for the operator to be well-defined.
  • Linearity gives Ψ(ξ^(K))=P_KΨ(ξ), linking truncated coefficient sequences to truncated solution expansions.
  • The truncation error is bounded using Poincaré’s inequality and Stechkin’s inequality.

Appendix D. Error During Training

Figures 12 and 13 track relative test errors during training for the linear elliptic and Poisson problems. Both show slight overfitting with more training samples when the reduced dimension is sufficiently large, attributed to the nonlinear neural-network parameterization of a linear map.

  • Appendix D. Error During Training: Figures 12 and 13 report relative test errors over training epochs for the linear elliptic and Poisson problems.The amount of training data varies across panels.
  • Appendix D. Error During Training: Both problems exhibit slight overfitting when more training samples are used and the reduced dimension is sufficiently large.This behavior is observed during the training process.
  • Appendix D. Error During Training: The paper attributes this overfitting to the true map being linear while the neural-network parameterization is highly nonlinear.The nonlinear parameterization is therefore more prone to overfitting larger amounts of data.
Loading 2005.03180v2…