PaperlayerView exact revision on arXiv

Source-linked AI summary

Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders

Kookjin Lee, Kevin Carlberg

arXiv:1812.08373v3math.NA

TL;DR

Linear-subspace ROMs struggle with slowly decaying Kolmogorov n-widths, especially in advection-dominated problems. This paper projects dynamics onto nonlinear manifolds through manifold Galerkin and manifold LSPG formulations and constructs those manifolds with convolutional autoencoders. On benchmark problems, the method outperforms optimal linear-subspace projections and approaches optimal nonlinear-manifold performance, while requiring more costly training and more hyperparameters.

  • Problem

    Slowly decaying Kolmogorov n-widths make low-dimensional linear-subspace ROMs inaccurate for many advection-dominated problems.

  • Method

    The paper uses minimum-residual manifold Galerkin and manifold LSPG projections with nonlinear manifolds computed from snapshot data by deep convolutional autoencoders.

  • Results

    The proposed methodology substantially lowers errors for low-dimensional reduced states, outperforming optimal linear-subspace projections and nearly achieving optimal nonlinear-manifold performance.

  • Takeaways & Limitations

    Nonlinear-manifold ROMs can overcome the intrinsic n-width limitations of linear trial subspaces on advection-dominated benchmark problems.

  • Takeaways & Limitations

    Training the deep convolutional autoencoder is more computationally expensive than computing POD singular vectors, and the method has significantly more architecture-related hyperparameters.

Abstract

from arXiv · show

Nearly all model-reduction techniques project the governing equations onto a linear subspace of the original state space. Such subspaces are typically computed using methods such as balanced truncation, rational interpolation, the reduced-basis method, and (balanced) POD. Unfortunately, restricting the state to evolve in a linear subspace imposes a fundamental limitation to the accuracy of the resulting reduced-order model (ROM). In particular, linear-subspace ROMs can be expected to produce low-dimensional models with high accuracy only if the problem admits a fast decaying Kolmogorov $n$-width (e.g., diffusion-dominated problems). Unfortunately, many problems of interest exhibit a slowly decaying Kolmogorov $n$-width (e.g., advection-dominated problems). To address this, we propose a novel framework for projecting dynamical systems onto nonlinear manifolds using minimum-residual formulations at the time-continuous and time-discrete levels; the former leads to manifold Galerkin projection, while the latter leads to manifold least-squares Petrov--Galerkin (LSPG) projection. We perform analyses that provide insight into the relationship between these proposed approaches and classical linear-subspace reduced-order models; we also derive a posteriori discrete-time error bounds for the proposed approaches. In addition, we propose a computationally practical approach for computing the nonlinear manifold, which is based on convolutional autoencoders from deep learning. Finally, we demonstrate the ability of the method to significantly outperform even the optimal linear-subspace ROM on benchmark advection-dominated problems, thereby demonstrating the method's ability to overcome the intrinsic $n$-width limitations of linear subspaces.

1. Introduction

Linear-subspace ROMs face accuracy limits on problems with slowly decaying Kolmogorov n-widths, motivating nonlinear-manifold projection and snapshot-trained convolutional autoencoders. The proposed methods are analyzed theoretically and outperform optimal linear-subspace projections on advection-dominated benchmarks.

  • Motivation: High-fidelity simulations can require months on thousands of computing cores, preventing their use in real-time or many-query scenarios.Projection-based ROMs address this burden through offline training and inexpensive online solution computation.
  • Motivation: Slowly decaying Kolmogorov n-widths in advection-dominated problems make low-dimensional linear-subspace ROMs inaccurate unless their dimension is substantially increased.Fast n-width decay can justify linear subspaces, whereas advection-dominated problems often retain large approximation error at the intrinsic solution-manifold dimension.
  • Nonlinear-manifold projection: The framework projects dynamical systems onto arbitrary nonlinear trial manifolds using minimum-residual formulations at time-continuous and time-discrete levels.These formulations yield manifold Galerkin and manifold LSPG projection, respectively; affine manifolds recover classical linear-subspace Galerkin and LSPG projection.
  • Autoencoder construction: The nonlinear manifold is computed from snapshot data alone using a deep convolutional autoencoder whose decoder maps reduced states back to high-dimensional states.The architecture is tailored to spatially distributed dynamical-system states and requires the same snapshot data as POD-based approaches.
  • Analysis and implementation: The methods include exact initial-condition satisfaction, quasi-Newton solvers for implicit integration, commutativity and equivalence analyses, and a posteriori discrete-time error bounds.These results connect manifold ROMs to classical methods and characterize their time-discrete behavior.
  • Numerical evidence: Numerical experiments on advection-dominated benchmarks outperform even optimal linear-subspace projections and approach optimal nonlinear-manifold performance.The experiments demonstrate that the methodology can overcome intrinsic n-width limitations of linear trial subspaces.

2. Full-order model

The full-order model is a parameterized dynamical system written as an ordinary differential equation, with states depending on time and parameters. Linear multistep time discretization converts the ODE into sequential algebraic residual equations.

  • Continuous model: The full-order model is a parameterized ODE with state x(t; µ), initial condition x0(µ), and velocity f(x,t,µ).The velocity may be linear or nonlinear in the state and may arise from semidiscretizing a PDE.
  • Time discretization: Linear multistep discretization produces algebraic equations rn(xn; µ) = 0 for time steps n = 1, ..., Nt.The resulting system is called the full-order O∆E.
  • Time discretization: The time-discrete residual depends on the time step, prior numerical states, and coefficients defining the selected linear multistep scheme.Implicit schemes occur when β0 ≠ 0; uniform time steps and fixed step number are assumed for simplicity.
  • Solution manifold: For a unique trajectory at each parameter instance, the intrinsic solution-manifold dimensionality is at most nµ + 1.The bound counts the nµ parameters and time as coordinates of the mapping to the state.

3. Model reduction on nonlinear manifolds

The paper formulates residual-minimizing reduced-order models on continuously differentiable nonlinear trial manifolds, yielding manifold Galerkin and manifold LSPG projections. The manifold is represented by a decoder from low-dimensional coordinates to the high-dimensional state, while classical affine trial subspaces arise as a special linear-decoder case.

  • Residual-minimizing ROMs: The framework minimizes time-continuous or time-discrete full-order-model residuals on nonlinear trial manifolds, producing manifold Galerkin and manifold LSPG ROMs.Manifold Galerkin minimizes the time-continuous residual; manifold LSPG minimizes the time-discrete residual.
  • Nonlinear trial manifolds: A decoder maps low-dimensional generalized coordinates to high-dimensional state approximations, and its Jacobian defines the manifold’s tangent space.The decoder is g: R^p → R^N, with Jacobian J defining the tangent space used by the projection.
  • Nonlinear trial manifolds: The manifold dimension must satisfy p ≥ p⋆ to represent the state exactly, where p⋆ denotes the intrinsic solution-manifold dimensionality.The intrinsic dimensionality is tied to the number of parameters plus time, while exact representation requires a sufficiently large manifold dimension.
  • Nonlinear trial manifolds: Initial-condition satisfaction requires generalized coordinates whose decoded state equals the initial full-order state.The paper proposes choosing initial coordinates and reference states so the decoder represents deviations from the reference state accurately.
  • Linear trial subspaces: A linear decoder recovers the classical affine trial-subspace setting, with a constant decoder Jacobian equal to the reduced-basis matrix.This special case connects the nonlinear-manifold formulation to standard linear-subspace ROMs.
  • Projection formulations: Manifold Galerkin orthogonally projects velocity onto the manifold tangent space, whereas manifold LSPG minimizes the time-discrete residual sequentially.Both formulations evolve low-dimensional generalized coordinates, and the time-discrete formulation can be posed with weighted norms, although this work uses the ℓ2 norm.
  • Scope: The formulation restricts attention to the ℓ2 norm, while weighted norms are left for future work, including possible hyper-reduction applications.This restriction applies to both manifold Galerkin and manifold LSPG residual minimization.
  • Projection formulations: The proposed projections do not require an encoder from the high-dimensional state to reduced coordinates, unlike alternative autoencoder-based Galerkin approaches.The paper also notes that encoding velocities can be kinematically inconsistent because velocities lie in the tangent space rather than generally on the manifold.

4. Analysis

The analysis establishes when nonlinear-manifold projections recover classical linear-subspace ROMs and when time discretization commutes with manifold Galerkin projection. It also gives limiting-equivalence conditions and a posteriori discrete-time error bounds, including sequential minimization by manifold LSPG.

  • Relations to linear-subspace ROMs: Affine trial manifolds recover classical LSPG projection, and orthogonal affine decoders additionally recover classical Galerkin projection.Thus, the nonlinear-manifold framework contains standard linear-subspace Galerkin and LSPG methods as special cases.
  • Relations to linear-subspace ROMs: The linear-decoder substitutions reproduce the classical Galerkin and LSPG residual systems under the stated basis conditions.For an orthogonal trial-basis matrix, the manifold Galerkin ODE uses Φ^T and is equivalent to the classical Galerkin ODE.
  • Time discretization and projection: For manifold Galerkin projection, time discretization and projection commute for affine manifolds or under twice-continuous differentiability, O(∆t) coordinate changes, and ∆t → 0.The result extends prior commutativity analysis from linear to nonlinear trial manifolds.
  • Time discretization and projection: The discretize-then-project formulation is equivalent to manifold Galerkin exactly when a suitable full-rank matrix sequence exists.The paper characterizes this equivalence through conditions on the test basis and residual representation.
  • Galerkin–LSPG equivalence: Manifold Galerkin and manifold LSPG are equivalent for affine manifolds with explicit schemes or ∆t → 0, and asymptotically for sufficiently smooth nonlinear manifolds.The nonlinear case requires twice-continuous differentiability, O(∆t) coordinate changes, and the limit ∆t → 0.
  • A posteriori error bounds: The manifold LSPG method sequentially minimizes the local contribution to the a posteriori discrete-time error bound.The bound assumes Lipschitz-continuous velocity and a sufficiently small time step; its first term is the new time-instance contribution and its second term is recursive.

5. Nonlinear trial manifold based on deep convolutional autoencoders

The paper constructs nonlinear trial manifolds with decoder mappings learned by deep convolutional autoencoders, tailored to spatially distributed dynamical-system states. These mappings support Deep Galerkin and Deep LSPG reduced-order models.

  • Nonlinear manifold construction: Deep convolutional autoencoders provide a neural-network decoder g: R^p → R^N for constructing the nonlinear trial manifold.The decoder maps generalized coordinates to approximations of the high-dimensional state.
  • Architecture rationale: Convolutional autoencoders are selected because they are expressive and scalable for spatially distributed states, unlike directly applying fully connected MLP autoencoders at large state dimensions.Fully connected MLP autoencoders can require extremely many parameters and substantial training data when n_x is large.
  • Autoencoder structure: An autoencoder learns an approximate identity by encoding a high-dimensional state into a low-dimensional code and decoding that code into a reconstructed state.The encoder maps R^n_x to R^n̂_x with n̂_x ≪ n_x, while the decoder maps the code back to R^n_x.
  • Network architecture: Convolutional layers use convolution operators with filter weights, whereas fully connected layers use affine weight-and-bias transformations followed by nonlinear activations.The decoder’s final layer has no nonlinear activation before inverse scaling and prolongation.
  • Network architecture: The encoder applies convolutional layers followed by fully connected layers, while the decoder reverses this pattern with fully connected and transposed-convolutional layers.Restriction and prolongation operators reshape states before and after the neural-network operations.
  • Resulting ROMs: Setting the manifold decoder g equal to the trained autoencoder decoder defines Deep Galerkin and Deep LSPG ROMs.These names correspond respectively to manifold Galerkin and manifold LSPG projection with this decoder choice.

6. Offline training

Offline training uses the same snapshot data as POD, standardizes the data, and trains the autoencoder with stochastic gradient descent and early stopping.

  • Snapshot collection: The offline training process uses precisely the same snapshot data and data-collection procedure employed by POD.Snapshots are obtained by solving the full-order model for training-parameter instances.
  • Parameter optimization: The autoencoder parameters are optimized with stochastic gradient descent using minibatching, adaptive learning rates, and early stopping.The optimization returns the trained encoder and decoder g.
  • Snapshot collection: Training first solves the full-order model for training parameters and forms the snapshot matrix W.The matrix contains time-state differences for each training parameter instance.
  • Data preprocessing: A scaling operator S is computed from the training data before defining the complete autoencoder.The restriction operator reformats vectors as tensors, while scaling improves numerical stability during training.
  • Comparison with POD: POD uses the same snapshot matrix W to compute a linear trial basis from the leading left singular vectors of its singular value decomposition.This contrasts snapshot-based POD with the autoencoder training procedure used for the nonlinear manifold.

7. Numerical experiments

The experiments compare nonlinear-manifold Deep Galerkin and Deep LSPG ROMs with POD-based linear-subspace ROMs on advection-dominated Burgers’ and chemically reacting-flow benchmarks. Across these tests, nonlinear-manifold ROMs achieve substantially lower errors, including near 0.1% accuracy at dimensions matching or only slightly exceeding the intrinsic solution-manifold dimension.

  • Experimental setup: The study evaluates POD–Galerkin, POD–LSPG, Deep Galerkin, and Deep LSPG ROMs on 1D Burgers’ equation and chemically reacting flow.The nonlinear methods use a deep convolutional decoder, while the POD methods use a POD basis; all methods are compared at matched reduced dimensions.
  • Experimental setup: The ROMs use a 10-layer convolutional autoencoder with a p-dimensional latent code, trained using ELU activations, an ℓ2 loss, and Adam optimization.The encoder has four convolutional layers and one fully connected layer; the decoder reverses this arrangement with transposed convolutions.
  • Error assessment: The experiments assess relative state error, POD and optimal-basis projection errors, and manifold projection error.The optimal-basis projection error provides an ℓ2-norm counterpart to the Kolmogorov p-width, while the POD projection error lower-bounds POD–Galerkin and POD–LSPG errors.
  • 1D Burgers’ equation: For Burgers’ equation, nonlinear-manifold ROMs produce extremely accurate solutions, whereas same-data linear-subspace ROMs exhibit significant errors.The reported comparison attributes this contrast to the ill-suitedness of linear trial subspaces for advection-dominated problems.
  • 1D Burgers’ equation: With Burgers’ intrinsic dimension p⋆=3, Deep LSPG reaches relative errors near 0.1% at p=5, while linear-subspace methods remain near 10%.The optimal linear-basis projection error is still above 0.1% at p=50, whereas the nonlinear method requires only p=p⋆+2 for sub-0.1% errors.
  • Chemically reacting flow: For chemically reacting flow, both Deep Galerkin and Deep LSPG achieve relative errors below 0.1% at p=p⋆=3, while the optimal linear-basis projection error is around 5%.The nonlinear-manifold ROMs also produce significantly lower temperature and H2 mass-fraction errors than the linear-subspace ROMs.

8. Conclusion

The work develops nonlinear-manifold Galerkin and LSPG ROMs and a convolutional-autoencoder strategy for constructing their manifolds. Numerical experiments show substantially lower errors than optimal linear-subspace projections, while higher training cost and future hyper-reduction needs remain.

  • The proposed manifold Galerkin and manifold LSPG techniques project dynamical-system models onto arbitrary continuously differentiable nonlinear manifolds.
  • A practical strategy computes representative low-dimensional nonlinear trial manifolds using a tailored convolutional autoencoder.
  • The numerical experiments compare Deep Galerkin and Deep LSPG against POD–Galerkin and POD–LSPG using advection-dominated benchmark problems, including chemically reacting flow.
  • Sub-0.1% errors require a reduced dimension only two above the intrinsic solution-manifold dimension in one case and exactly equal to it in the other.
  • The method's training process is costlier than computing snapshot singular vectors and introduces significantly more autoencoder-architecture hyperparameters than classical methods.
  • Future work includes integrating the projections into a full hyper-reduction framework to realize computational cost savings and applying them to truly large-scale dynamical-system models.

Appendix A. Stochastic gradient descent for autoencoder training

The appendix describes autoencoder training with stochastic gradient descent, minibatches, adaptive learning rates, validation-based early stopping, and shuffled training and validation snapshots.

  • Autoencoder parameters are computed using stochastic gradient descent with minibatching and early stopping.
  • The snapshot matrix is randomly shuffled into training and validation snapshots before training.
  • The autoencoder parameters approximately minimize empirical reconstruction risk over the training data.
  • Minibatch gradients are sample means over selected batches and are computed through automatic differentiation, or backpropagation.
  • Adaptive learning-rate strategies modify parameter updates using parameter-specific learning rates, with AdaGrad, RMSProp, and Adam given as examples.
  • Early stopping terminates optimization when validation loss fails to decrease for a specified number of epochs, acting as a regularizer and an iteration-count hyperparameter.

Appendix B. Basics of convolutional layers

The appendix introduces convolutional-layer operations using feature maps and filter banks, with convolution, nonlinear activation, and pooling forming the layer computation.

  • Convolutional-layer units are organized as feature maps connected to local patches of preceding feature maps through discrete convolution.
  • Feature maps are represented as three-dimensional tensors, while filter banks are represented as four-dimensional tensors.
  • A convolutional layer applies a filter bank and bias, followed by nonlinear activation and pooling.
  • The stride controls downsampling, and strides greater than one reduce the next feature map's dimensions in each direction.
  • Filter-bank weights and biases are learnable parameters, whereas kernel lengths, filter counts, and stride are hyperparameters.

Appendix C. Additional numerical experiments: extrapolation and interpolation in time

Additional experiments assess whether the proposed Deep Galerkin and Deep LSPG ROMs extrapolate and interpolate in time, and examine early stopping using 1D Burgers' equation.

  • The experiments evaluate temporal extrapolation and interpolation for Deep Galerkin and Deep LSPG ROMs.
  • The experiments also assess the effect of early stopping on method performance.
  • All experiments use the 1D Burgers' equation setup from Section 7.1 unless otherwise specified.

Appendix C.1. Time extrapolation

This experiment evaluates time extrapolation using snapshots from the initial time instances and compares Deep and POD ROMs at p = 5. The deep-manifold methods become more accurate than the POD methods as more training snapshots are included.

  • For nt ≥300, Deep Galerkin and Deep LSPG are more accurate than POD–Galerkin and POD–LSPG at the tested online parameters.
  • Around 0.1% relative error is achieved by Deep LSPG for nt ≥400.
  • The figure reports relative errors for POD–Galerkin, POD–LSPG, Deep Galerkin, and Deep LSPG ROMs of dimension p = 5.

Appendix C.2. Time interpolation

This experiment evaluates time interpolation using equally spaced snapshots spanning the full time interval. The proposed deep ROMs remain more accurate than the POD ROMs even with substantially fewer snapshots.

  • Snapshots are collected at equally spaced time instances, including the final time t_Nt, with nt ∈ {10, 20, 50, 100, 140, 250, 500}.
  • POD–Galerkin and POD–LSPG errors are almost entirely insensitive to snapshot count for nt ≥10.
  • Deep Galerkin and Deep LSPG are insensitive to snapshot count for nt ≥100.
  • For nt ∈ {10, 20, 50}, the deep ROMs have larger errors than for nt ≥100 but remain more accurate than the POD ROMs.
  • The figure compares relative errors of the four ROM types at online parameters µ1 test and µ2 test for dimension p = 5.

Appendix C.3. Early-stopping study

The early-stopping study trains autoencoders with and without early stopping at reduced dimension p = 30, while the appendix also documents decoder-cost estimation and layer-operation counts.

  • The study compares autoencoder training with and without early stopping at reduced dimension p = 30.Without early stopping, training runs for at most 1000 epochs and saves network parameters at each epoch.
  • Table C.2 reports relative ROM errors at online parameter µ1 test for offline training performed with and without early stopping.
  • A fully connected layer evaluates a matrix–vector product followed by an element-wise activation, with cost 2p_i(p_{i−1}+1)+c_φp_i flops.
  • Convolutional layers apply discrete convolutions followed by element-wise nonlinear activations.
  • Restrictor and prolongator operations only reshape snapshot matrices, while scaling and inverse scaling each contribute to the stated 2N-flop cost.
  • Decoder costs are estimated for latent dimension p = 5 across convolution filter sizes k = {25, 16, 9, 5, 3} and multiple filter configurations.
  • The cost analysis separates fully connected and transposed-convolutional layers and compares totals with linear decoders at p = 85 and p = 5.
Loading 1812.08373v3…