PaperlayerView exact revision on arXiv

Source-linked AI summary

A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder

Youngkyu Kim, Youngsoo Choi, David Widemann, Tarek Zohdi

arXiv:2009.11990v2math.NAcs.LGcs.NE

TL;DR

The paper addresses the limited representability of LS-ROMs for advection-dominated and sharp-gradient solutions. It develops a physics-informed NM-ROM using a shallow masked autoencoder and hyper-reduction while retaining existing numerical methods. In Burgers’ experiments, the approach achieves accurate reduced simulations with speedups exceeding 2 in 1D and about 12 in 2D, alongside posteriori error bounds.

  • Problem

    LS-ROMs struggle with advection-dominated or sharp-gradient solutions because their intrinsic solution spaces do not have small Kolmogorov n-width.

  • Method

    The paper trains a shallow masked autoencoder, uses its decoder as a nonlinear manifold, retains existing numerical methods, and applies hyper-reduction to nonlinear terms.

  • Results

    The NM-LSPG-HR method achieves maximum relative error around 1% with speedup higher than 2 for 1D Burgers’ and error below 1% with speedup around 12 for 2D Burgers’.

  • Takeaways & Limitations

    The results support accurate and efficient NM-ROM simulations for the tested 1D and 2D Burgers’ advection-dominated data.

  • Takeaways & Limitations

    Future application to more complex simulations requires balancing network size sufficient to capture data variance against speed faster than the FOM, while residual-basis and sample-point sizes were selected from test results.

Abstract

from arXiv · show

Traditional linear subspace reduced order models (LS-ROMs) are able to accelerate physical simulations, in which the intrinsic solution space falls into a subspace with a small dimension, i.e., the solution space has a small Kolmogorov n-width. However, for physical phenomena not of this type, e.g., any advection-dominated flow phenomena, such as in traffic flow, atmospheric flows, and air flow over vehicles, a low-dimensional linear subspace poorly approximates the solution. To address cases such as these, we have developed a fast and accurate physics-informed neural network ROM, namely nonlinear manifold ROM (NM-ROM), which can better approximate high-fidelity model solutions with a smaller latent space dimension than the LS-ROMs. Our method takes advantage of the existing numerical methods that are used to solve the corresponding full order models. The efficiency is achieved by developing a hyper-reduction technique in the context of the NM-ROM. Numerical results show that neural networks can learn a more efficient latent space representation on advection-dominated data from 1D and 2D Burgers' equations. A speedup of up to 2.6 for 1D Burgers' and a speedup of 11.7 for 2D Burgers' equations are achieved with an appropriate treatment of the nonlinear terms through a hyper-reduction technique. Finally, a posteriori error bounds for the NM-ROMs are derived that take account of the hyper-reduced operators.

1. Introduction.

Traditional LS-ROMs efficiently represent solutions with small Kolmogorov n-width, but struggle with advection-dominated or sharp-gradient phenomena. The paper develops NM-ROMs using nonlinear manifolds and hyper-reduction to improve representability while retaining speed and accuracy.

  • Linear subspace ROMs: LS-ROMs compress simulation data into a linear basis and substitute that representation into discretized governing equations to reduce unknowns.They exploit both governing equations and simulation data.
  • Limitations of LS-ROMs: LS-ROMs poorly represent advection-dominated or sharp-gradient solutions because these problems lack small Kolmogorov n-width.Examples include high-Reynolds-number hyperbolic equations, Boltzmann transport, and traffic-flow simulations.
  • Prior approaches: Existing remedies enhance linear subspaces or replace them with nonlinear manifolds, but linear-subspace treatments are problem-specific and require prior knowledge such as advection direction.This limitation motivates more general nonlinear-manifold representations.
  • Paper contribution: The paper presents NM-ROMs that use a trained shallow masked autoencoder decoder as a nonlinear manifold solution representation.The approach differs from prior neural ROMs by using a shallow masked rather than deep convolutional autoencoder.
  • Paper contribution: NM-ROMs retain existing PDE/ODE numerical methods while introducing hyper-reduction to address nonlinear terms that otherwise scale with full-order-model size.The paper develops NM-LSPG-HR and NM-Galerkin-HR formulations.

2. Background.

The paper formulates parameterized nonlinear ODEs, discretizes them with implicit time integration, and reviews linear-subspace Galerkin and LSPG reduced-order projections. POD constructs the linear basis, while hyper-reduction is required to obtain speedups when nonlinear terms scale with the full-order dimension.

  • 2.1. Full order model.: A parameterized nonlinear ODE represents the spatially semi-discretized governing PDE, with time-dependent state x(t; µ), velocity f, initial state x0, and parameters µ.
  • 2.1. Full order model.: Backward Euler uses the update x_n − x_{n−1} = Δt f_n and implies that nonlinear-term snapshots lie in the span of solution snapshots.
  • 2.2.2. Linear subspace Galerkin projection.: LS-Galerkin approximates the state in a reference solution plus a linear subspace, then projects the continuous residual to obtain an n_s-dimensional reduced ODE.
  • 2.2.1. Linear subspace solution representation.: POD obtains an n_s-dimensional orthonormal basis from a truncated SVD of the FOM solution snapshot matrix and minimizes the snapshot reconstruction error in the Frobenius norm.
  • 2.2.3. Linear subspace least-squares Petrov–Galerkin projection.: LSPG replaces the fully discretized state with the trial subspace representation and minimizes the squared residual norm at each time step, typically using Gauss–Newton.
  • Nonlinear manifold ROM.: The paper introduces a projection-based ROM with a nonlinear manifold representation, neural-network construction, Galerkin and LSPG projections, and NM-ROM hyper-reduction.

3. Nonlinear manifold reduced order model (NM-ROM).

The NM-ROM uses a shallow masked autoencoder to represent full-order solutions on a nonlinear manifold with a low-dimensional latent space. Masking sparsifies decoder computations so hyper-reduction can reduce the cost of evaluating nonlinear terms.

  • 3.1. Nonlinear manifold solution representation.: The NM-ROM represents solutions as a nonlinear manifold S = {g(ˆv) | ˆv ∈ R^ns}, where g maps latent coordinates to the full-order space and ns ≪ Ns.The initial generalized coordinate is obtained with an approximate inverse encoder h.
  • 3.2. Shallow masked autoencoder.: The autoencoder encodes a high-dimensional input into a low-dimensional latent vector and decodes it into a reconstructed full-order state.It is trained to reconstruct its input, learning salient features through compression and reconstruction.
  • 3.2. Shallow masked autoencoder.: A single-hidden-layer decoder is used because its value and Jacobian are evaluated repeatedly during ROM computations.The shallow architecture limits the matrix-matrix multiplications required by deeper decoder Jacobians.
  • 3.2. Shallow masked autoencoder.: The decoder output layer applies a binary sparsity mask so sampled outputs require only the corresponding active paths through the hidden layer.The mask matrix S is applied element-wise to the output-layer weights, producing a sparser decoder structure.
  • 3.2. Shallow masked autoencoder.: The number of learnable parameters depends on hidden-layer widths, latent dimension, and mask sparsity, but these hyper-parameters cannot be determined a priori.Too few parameters can limit manifold representation, whereas too many can cause over-fitting and poor generalization to unseen data.
  • 3.3–3.4. Hyper-reduction.: Hyper-reduction evaluates residuals at sample points and restricts decoder and Jacobian computations to active paths, with costs scaling piecewise-linearly with sample count.Without hyper-reduction, nonlinear terms and Jacobians still scale with the full-order model size.

4. Hyper-reduction.

The paper hyper-reduces nonlinear residuals and decoder evaluations by sampling selected entries and restricting computation to relevant basis rows and decoder active paths. These constructions yield reduced formulations whose costs can scale with the number of sample points rather than the full-order size.

  • Residual hyper-reduction: Gappy POD approximates nonlinear residuals using a residual basis and selected sample points, reducing nonlinear-term update complexity.GNAT-SNS constructs a residual basis from solution snapshots and solves a least-squares problem over sampled entries.
  • Residual hyper-reduction: The oblique projection P = Φr(ZT Φr)†ZT reconstructs sampled residual information while allowing oversampling with nr < nz.The sampling matrix is represented operationally through selected indices and corresponding rows rather than explicitly constructed.
  • Linear-subspace ROMs: LS-Galerkin-HR and LS-LSPG-HR replace their residuals with the hyper-reduced approximation and retain reduced systems with ns unknowns and ns equations.The selected basis rows and nonlinear-term entries can be precomputed or evaluated only at the sampling points.
  • Nonlinear-manifold ROMs: NM-ROMs contain nonlinearities in both the governing equations and decoder, so hyper-reduction also restricts decoder and Jacobian evaluation to sampled outputs and active paths.The sampled decoder Jacobian must be updated as the latent coordinates change, whereas selected nonlinear-term entries alone are computed from the governing equations.
  • Nonlinear-manifold ROMs: NM-LSPG-HR evaluates residuals at hyper-reduction indices, with decoder and Jacobian costs scaling piecewise-linearly with the number of sample points.The method computes only the active paths needed for outputs contributing to selected residual elements.
  • Computational cost: NM-LSPG costs are O(mbf), whereas NM-LSPG-HR costs are O(zbf) + O(fz2), so speed-up can result when z2 < m.Here m is the FOM size, f the ROM size, z the subnet-output size, and b the hidden-layer width per sparse-decoder output.

5. Error analysis.

The error analysis defines hyper-reduced NM-ROM residuals by substituting the reconstructed solution into the full residual functions and introduces assumptions for the multi-step scheme.

  • Error-bound setup: Theorem 5.1 analyzes NM-Galerkin-HR and NM-LSPG-HR using a sampling matrix Z, an oblique projection P, and the nonlinear residual term ˜rn.The residual is formed from the approximate state xref + g(ˆxn) within the full residual functions.
  • Assumptions: The multi-step error analysis uses coefficients αj and βj for the scheme, with parameters γ1 and γ2 satisfying 0 < γ1, γ2 ≤ 1.These coefficients define the particular linear multi-step method considered in the bound.
  • Error-bound setup: The approximate NM-ROM state is represented as xref + g(ˆxn), while xn denotes the corresponding FOM solution.This distinction underlies the residual-based comparison in the error analysis.

Subtracting Eq. (5.3) from Eq. (5.4) gives

The derivation compares the FOM residual with the hyper-reduced NM-ROM residual and uses projection relations, Lipschitz continuity, and a time-step condition to establish the error bounds. For NM-LSPG-HR, the computed solution sequentially minimizes the resulting error bound.

  • Residual comparison: The proof rewrites the difference between the FOM and approximate residuals before applying the reverse triangle inequality.The derivation separates projected residual terms to compare the exact and approximate states.
  • Projection estimate: Projection relations express the nonlinear-term difference through γ2, the projection norm, and the difference between f evaluated at the exact and approximate states.The analysis assumes 0 < γ1 ≤ 1 and 0 < γ2 ≤ 1.
  • Time-step condition: If f is Lipschitz continuous with constant L and ∆t < γ1|α0|/(γ2|β0|L), the time-step restriction supports the subsequent error estimate.The condition links the time-step size to the multi-step coefficients, projection factors, and Lipschitz constant.
  • Error bounds: Triangle inequality and Lipschitz continuity are combined with earlier relations to prove the NM-Galerkin-HR error bound.The proof explicitly combines Eqs. (5.5) and (5.6).
  • Error bounds: NM-LSPG-HR solutions satisfy sequential minimization of the error bound derived for the hyper-reduced method.This follows from minimizing the projected nonlinear residual term in the NM-LSPG-HR formulation.
  • Numerical context: The numerical study evaluates NM-ROM accuracy and speed-up on parameterized 1D inviscid and 2D viscous advection-dominated Burgers equations.The ROMs are trained on solution snapshots from training parameters and used for prediction at an excluded parameter.

6. Numerical results.

Numerical tests on 1D inviscid and 2D viscous Burgers’ equations show that NM-ROMs represent advection-dominated solutions accurately with low latent dimension, while hyper-reduction is essential for speed-up.

  • 1D inviscid Burgers’ equation: NM-ROM relative errors are lower than LS-ROM errors for 1D Burgers’ equation, even below the LS-ROM projection-error lower bounds.The comparison uses a target parameter excluded from training.
  • 1D inviscid Burgers’ equation: 1D LS-ROMs are about 5 to 6 times faster than FOM with 6% maximum relative error, whereas NM-ROMs achieve less than 1% maximum relative error.Hyper-reduction improves LS-ROM speed-up to 9 to 10 times, but NM-ROMs require hyper-reduction to obtain speed-up.
  • Generalization capability: NM-LSPG and NM-LSPG-HR accuracy deteriorates gradually outside the training parameter domain, implying an application-dependent trust region.For the 1D test, a 10% maximum-relative-error criterion gives trust region [0.6, 1.2].
  • 2D viscous Burgers’ equation: Re = 10000 produces slowly decaying singular values and sharp gradients in the 2D Burgers’ solution.The high-Reynolds-number case is advection-dominated.
  • 2D viscous Burgers’ equation: Without hyper-reduction, NM-Galerkin and LS-ROMs do not achieve speed-up over the 2D FOM, making hyper-reduction essential for considerable speed-up with reasonable accuracy.The reported FOM wall time is 140.67 seconds.
  • 2D viscous Burgers’ equation: The 2D NM-LSPG-HR achieves around 1% relative error and speed-up higher than 11, while LS-LSPG-HR has relative errors around 37%.Both methods use reduced dimension ns = 5, with varying residual bases and samples.
  • 2D viscous Burgers’ equation: NM-LSPG-HR solutions agree well with FOM solutions and can match NM-LSPG accuracy for some small residual-basis and residual-sample combinations.In the 2D test, ns = 5, with 55 residual-basis dimensions and 58 residual samples for NM-LSPG-HR.

7. Discussion & conclusion.

The NM-LSPG-HR method combines a sparse shallow decoder with hyper-reduction to achieve accurate, accelerated simulations, while the authors identify scaling and parameter-selection considerations for future work.

  • Results: Less than 1% maximum relative error and around 12 speed-up were achieved for the 2D case.This speed-up was better than the reported 1D result.
  • Scaling: Higher speed-ups are expected as the FOM size increases because the computational-cost difference between FOM and NM-LSPG-HR grows.The computational-cost trend is illustrated in Fig. 17.
  • Mechanism: The NM-LSPG-HR speed-up uses a sparse shallow decoder as the nonlinear manifold and efficient hyper-reduction computed through a subnet.
  • Future work: Future work includes balancing shallow-network size against runtime for more complex simulations.The network must capture data variance while remaining faster than the FOM.
  • Future work: The proper residual-basis size and number of sample points were selected using test results rather than determined a priori.This issue applies to both NM-LSPG-HR and LS-LSPG-HR.

Appendix A. Time integrators.

Time integration of the governing equation produces a nonlinear algebraic residual system that must be solved for the numerical state at each time instance.

  • Time integration: Applying a linear k-step method yields a nonlinear algebraic residual function for the numerical solution x_n ∈ R^N_s at each time instance.
  • Time integration: The residual system is the discrete problem generated when a time integrator is applied to the governing equation.
  • Time integration: The numerical state is obtained by solving the residual function at each time instance.

A.1. The linear multistep methods.

The appendix describes linear multistep integrators, including Euler, Adams–Moulton, Adams–Bashforth, and BDF methods, through their nonlinear systems and residual functions.

  • Linear multistep methods: Linear multistep methods use k(t_n) previous steps, with k(t_n) ≤ n, to advance the numerical solution.
  • Linear multistep methods: The listed linear multistep methods include one-step Euler, implicit Adams–Moulton, explicit Adams–Bashforth, and Backward Differentiation Formulas.
  • Adams–Bashforth: The second-order Adams–Bashforth method advances x_n by solving a nonlinear system and defining its residual function.
  • Adams–Moulton: The second-order Adams–Moulton method similarly advances x_n through a nonlinear system and its residual function.
  • BDF and midpoint methods: The two-step BDF and midpoint methods are also presented as time-integration schemes for advancing the governing equation.

A.2. The midpoint Runge–Kutta method.

The reduced-order computational setup distinguishes FOM and ROM sizes and specifies a small sampled index set, including neighboring indices required by local models.

  • Notation: The FOM and ROM sizes are denoted by m and f, respectively.
  • Sampling: Local-information models require selected hyper-reduction indices together with their neighbors.
  • Sampling: The total number of indices is z, assumed to satisfy z ≪ m, z^2 < m, and z > f.An example given is z = 10f.

Appendix B. Computational costs.

The appendix derives computational costs for a shallow single-hidden-layer decoder and shows how hyper-reduction replaces full output sampling with z selected points.

  • The decoder and its Jacobian have costs O(mb)+O(M2f) and O(fM2)+O(mbf), respectively.The residual costs O(m), while its Jacobian costs O(fm) because of local model connectivity.
  • Hyper-reduction reduces the hidden-to-output weight matrix size from M2 to βM2 and replaces m with z in decoder evaluations.Here β ranges from z/m in the best case to 1 in the worst case, based on selected-point placement.
  • The resulting hyper-reduced decoder and Jacobian costs are O(zb)+O(βM2f) and O(fβM2)+O(zbf), respectively.

B.2. Computational costs of NM-LSPG-HR.

NM-LSPG-HR combines hyper-reduced decoder evaluations with residual and Gauss–Newton computations, yielding costs that depend on the sampled dimension z and reduction factor β.

  • NM-LSPG-HR has total per-time-step cost O(fβM2)+O(zbf)+O(fz2).The terms arise from hyper-reduced decoder/Jacobian work, residual-related work, and Gauss–Newton updates after factoring out iteration count.
  • Under M2 ≈ δbm, the NM-LSPG-HR cost becomes O(fβδbm)+O(zbf)+O(fz2).
  • For the best case β = z/m, the cost is O(fzδb)+O(zbf)+O(fz2), and assuming δb < b it reduces to O(zbf)+O(fz2).
  • For the worst case β = 1, the cost is O(fδbm)+O(zbf)+O(fz2).
  • Replacing the decoder with Φx̂ gives cost O(mf) and zero Jacobian-evaluation cost, while residual and Gauss–Newton costs remain unchanged from NM-LSPG.

B.3. Computational costs of LS-LSPG.

The appendix gives the LS-LSPG per-time-step cost and describes hyper-reduction through a basis matrix formed by selecting z rows of the full basis.

  • LS-LSPG has total computational cost O(f2m) for each time step.The number of Gauss–Newton iterations is factored out under the assumption that it has the same order for the governing equation.
  • For LS-LSPG-HR, the reduced basis matrix ΦHR has size z by f and is defined as ΦHR := ZT Φ.
  • The LS-LSPG-HR basis evaluation costs O(zf), and its Jacobian evaluation cost is zero.
Loading 2009.11990v2…