Nonlinear model reduction for transport-dominated problems

1.1. Model reduction

• Model reduction targets repeated simulations in design, control, optimization, uncertainty quantification, and inverse problems by replacing costly full models with low-dimensional reduced models.It exploits structured solution variability and permits substantial offline preprocessing when online simulations are inexpensive enough to amortize that cost.Source: §1.1 · p. 2

1.2. Linear model reduction and its limitation

• Classical linear model reduction represents a solution manifold with one global low-dimensional vector space, even when the resulting reduced dynamics are nonlinear.The term linear refers to the reduced-solution ansatz, which is a linear combination of basis functions, not necessarily to the governing dynamics.Source: §1.2 · p. 3
• For transport-dominated problems, linear approximation errors can decay as slowly as O(1/√n), versus exponential O(e^-n) for some diffusion-dominated problems, creating the Kolmogorov barrier.Traveling wave fronts and phase transitions produce solution manifolds that may be rough, curved, or otherwise poorly captured by low-dimensional vector spaces.Source: §1.2 · p. 3

1.3. Nonlinear model reduction

• Nonlinear model reduction allows reduced solutions to depend nonlinearly on their degrees of freedom, potentially representing transport-dominated solution manifolds with fewer degrees of freedom.Because nonlinear trial manifolds lack a global linear projection and associated orthogonality principle, their reduced formulations, analysis, and numerical solvers must be revisited.Source: §1.3 · pp. 4–5

1.4. Outline of the paper

• The survey organizes nonlinear model reduction around nonlinear transformations, online reduced-space adaptation, and generic nonlinear parametrizations paired with instantaneous residual minimization.It first reviews linear methods and their transport limitations, then develops these three categories before concluding with outlook.Source: §1.4 · p. 5

2.1. Full model

• The full model represents PDE solutions through N coefficients and derives their dynamics as a semi-discrete system of ordinary differential equations.The formulation covers time- and parameter-dependent PDEs with low-dimensional spatial domains and moderately dimensional parameter domains.Source: §2.1 · pp. 6–7

2.2. Overview of linear model reduction

• Linear model reduction seeks a fixed subspace V of dimension n≪N whose basis functions linearly approximate the solution manifold across all times and parameters.Reduced solutions are computed using n coefficients rather than the N coefficients required by the full-model representation.Source: §2.2 · pp. 7–8

2.3. The three steps of the offline phase of linear model reduction

• The offline workflow collects full-model snapshots, trains the reduced space, and constructs coefficient dynamics that are later simulated in the online phase.Training parameters may be selected incrementally by greedy procedures using error indicators and a posteriori error estimation.Source: §2.3 · p. 8, §2.3.1 · p. 9

2.3.2. Offline Step 2: Training a reduced space

• Proper orthogonal decomposition constructs the reduced basis from leading left singular vectors of a snapshot matrix, while alternative bases use greedy, interpolation-based, or balancing techniques.The snapshot matrix contains full-model states sampled over selected training parameters and discrete time points.Source: §2.3.2 · pp. 9–10

2.3.3. Offline step 3: Constructing a reduced model

• Galerkin or Petrov–Galerkin projection yields reduced ODEs for the n coefficients; Petrov–Galerkin formulations are important for stability with non-symmetric PDE operators.Although the ansatz is linear in the reduced state, the reduced right-hand side can remain nonlinear, so linear reduction applies to nonlinear full models.Source: §2.3.3 · pp. 10–11

2.4. Online computations, the lifting bottleneck, and empirical interpolation

• General online evaluations lift reduced states, evaluate all N full-model components, and project back, creating the lifting bottleneck; empirical interpolation instead evaluates only selected components.Discrete empirical interpolation selects interpolation indices offline and reconstructs the reduced right-hand side from those sampled values.Source: §2.4 · p. 11, §2.4.2 · pp. 12–13, §2.4.3 · p. 13

2.4.3. Empirical interpolation

• Empirical interpolation can bypass the lifting bottleneck by reducing online right-hand-side evaluations from N component functions to n when components are independently evaluable.Using the empirically interpolated right-hand side can provide runtime speedups even when projection-based evaluation does not.Source: §2.4.3 · p. 14
• Interpolation-point selection is central to empirical-interpolation accuracy, with greedy methods, QDEIM, and oversampling strategies addressing sampling quality and perturbation-induced instability.QDEIM uses QR pivoting and can yield sharper error bounds than greedy selection; randomized oversampling mitigates instabilities caused by noise or perturbations.Source: §2.4.3 · pp. 14–15

2.5.1. Diffusion problem: heat equation

• For the heat equation, the first n=20 left-singular vectors approximate the snapshots to double precision, despite full dimension N=1024, indicating effective low-dimensional linear reduction.The rapidly decaying singular values are a heuristic for expected reduction, not a guarantee for solutions outside the snapshot matrix.Source: §2.5.1 · p. 16

2.5.2. Transport problem: linear advection equation

• For linear advection, n≥150 modes are needed for double-precision snapshot representation, because translating a localized Gaussian produces shifted copies that a vector space represents inefficiently.The snapshot manifold is one-dimensional with fixed advection speed, yet its approximately algebraic singular-value decay makes linear reduction much less efficient than for diffusion.Source: §2.5.2 · pp. 16–17

3.1. Best linear approximations in vector spaces: the Kolmogorov 𝑛-width

• The Kolmogorov n-width measures the smallest worst-case approximation error over all n-dimensional subspaces, thereby characterizing the fundamental limit of linear reduced spaces.In a Hilbert space, best approximation is given by orthogonal projection, which is linear; the width is obtained by optimizing this worst-case error over subspaces.Source: §3.1 · pp. 17–18

3.2. Fast and slow decay of Kolmogorov 𝑛-width and Kolmogorov barrier

• Fast n-width decay can make low-dimensional linear reduction effective, whereas slow decay may require an inefficiently large reduced dimension, creating the Kolmogorov barrier.Diffusion-dominated problems often have rapidly decaying widths, including exponential decay for certain one-parameter elliptic problems; greedy reduced spaces can attain comparable decay up to constants.Source: §3.2 · pp. 18–19

3.3. Slow 𝑛-width decay for linear advection problem

• Transport-dominated problems with moving coherent features lead to slowly decaying n-widths, fundamentally limiting linear model reduction.For the linear advection solution manifold of step functions, the L2 n-width cannot decay faster than 1/√n, much slower than the exponential decay available for the cited diffusion-dominated problem; smoother initial data can improve the decay rate.Source: §3.3 · pp. 19–20

3.4. Kolmogorov n-width and linear control systems

• For asymptotically stable linear time-invariant control systems, Kolmogorov n-widths coincide exactly with Hankel singular values, although realizing the corresponding approximation quality with an explicit reduced model is nontrivial.The widths are therefore computable through the system’s Hankel operator, while optimal Hankel norm approximation addresses construction of a reduced model realizing that quality.Source: §3.4 · pp. 20–21
• Kolmogorov n-widths depend critically on the approximation space and norm, so Wasserstein metrics may provide different insights for transport-dominated problems.Parameter-dependent projection matrices can improve widths in parameterized systems, while standard Lp norms may provide an incomplete picture.Source: §3.4 · pp. 21–22

4.1.1. Parametrizations

• A parametrization maps an n-dimensional weight vector to a function, with the weights serving as reduced or latent states that are computed online through reduced dynamics.Linearity concerns dependence on the weight vector, not spatial dependence; in linear reduction, the resulting reduced space remains fixed while the weights vary with time and parameters.Source: §4.1.1 · p. 22, §4.1.2 · p. 23

4.1.3. Nonlinear parametrizations

• Nonlinear parametrizations replace fixed vector-space approximations with trial manifolds that may represent transport-dominated solutions efficiently using fewer weights.Their nonlinear dependence on the weight vector can circumvent the Kolmogorov barrier by allowing more general trial sets than linear reduced spaces.Source: §4.1 · p. 22, §4.1.3 · p. 23
• Nonlinear parametrizations can adapt the underlying representation itself, so weights may change feature shapes or basis functions rather than only coefficients of a fixed expansion.This includes representation-learning perspectives such as neural networks with nonlinear hidden layers.Source: §4.1.3 · p. 24

4.1.4. Example: Nonlinear parametrizations and linear advection

• For linear advection, nonlinear parametrization represents translated step functions exactly with n=2 weights, whereas linear-parametrization error decays no faster than 1/√n in L2.The nonlinear representation composes translation with feature superposition instead of approximating motion through many fixed basis functions.Source: §4.1.4 · p. 25

4.2. Extensions of Kolmogorov 𝑛-width to nonlinear parametrizations

• Nonlinear width concepts assess encoder–decoder approximation error, while stable manifold widths additionally require regularity such as Lipschitz bounds for numerical stability.Sensing numbers restrict the encoder to n linear functionals and can achieve faster decay than linear approximations for transported step functions.Source: §4.2 · p. 26

4.3.1. Offline and online weights

• Offline weights are fixed or trained from snapshots, whereas online weights act as low-dimensional latent states that evolve with time and parameters.Trained nonlinear parametrizations aim to make the online weight dimension n small while retaining expressive approximations.Source: §4.3.1 · pp. 26–27

4.3.2. Examples of nonlinear parametrizations with offline–online weights

• Examples combine polynomial dependence, learned transformations, adaptive representations, and neural networks, with offline weights defining architectures or features and online weights selecting coefficients or transformations.A transformation such as T(α,x)=x−α models translation through an online shift while offline training learns basis functions and transformation parameters.Source: §4.3.2 · pp. 27–28

4.4.1. Overview of elements of nonlinear model reduction

• Successful nonlinear model reduction requires expressive parametrizations, compatible reduced dynamics for evolving online weights, and efficient online evaluation and numerical solvers.Reducing degrees of freedom from N to n is insufficient unless the reduced model is computationally advantageous; error estimation and certification remain an additional element whose nonlinear formulations are still developing.Source: §4.4 · p. 28, §4.4.1 · pp. 28–29

4.4.3. Element 2: Reduced dynamics

• Reduced dynamics must remain compatible with nonlinear parametrizations, whose regularity and conditioning critically affect robustness, including where the parametrization becomes poorly conditioned.Source: §4.4.3 · p. 30

4.4.4. Element 3: Efficient online evaluation strategies and solvers

• Online solution advances the reduced weights through time or solves the steady reduced algebraic problem using time integrators, nonlinear-system iterations, and Jacobian-like solves when needed.Source: §4.4.4 · p. 30
• Nonlinear reduced dynamics often frame each time step as optimization, while variational formulations require online quadrature of parametrization-dependent inner products, norms, and derivatives.Source: §4.4.4 · p. 30
• Since these quantities depend nonlinearly on the weights, linear-style offline assembly is generally unavailable, so reducing degrees of freedom from N to n alone may not reduce online costs.Source: §4.4.4 · p. 31

5.1. Model reduction with transformation-based methods

• Transformation-based methods make solution manifolds linearly reducible by applying physically motivated or learned transformations to solutions, basis functions, or more general transformed modes.The two main formulations act on transformed full-model solutions or on reduced basis functions; their combination with linear reduction yields feature-wise, physically interpretable models.Source: §5 · p. 31, §5.1 · pp. 31–32

5.2. Literature review of transformation-based methods

• Symmetry, optimal-transport, registration, Lagrangian, and shock-alignment methods separate or align moving structures before applying linear reduction, while shifted POD represents multiple independently transported features.Group-action methods can struggle with multiple velocities; shifted POD assigns each low-rank component its own data-inferred shift, whereas registration smooths parameter dependence by aligning features.Source: §5.2.1 · p. 32, §5.2.2 · p. 33, §5.2.3 · p. 33

5.3.1. Shifted POD: Motivational example

• For one-dimensional advection, optimal linear POD uses Fourier modes unrelated to the initial profile, explaining oscillatory transport-dominated reduced models and motivating shifted POD.The advection solution is a translated initial function, while its Kolmogorov widths decay only polynomially.Source: §5.3.1 · p. 35

5.3.2. Shifted POD: Transformed modes

• Shifted POD introduces transformation operators and path variables as online weights, jointly optimizing modes, coefficients, and paths to represent distinct moving waves.The framework permits operator families beyond shifts, subject to continuity, invariance, and boundedness conditions; solvability requires suitable functional-analytic assumptions.Source: §5.3.2 · pp. 35–36

5.3.3. Shifted POD: Computational aspects for the optimization problem

• The transformed POD optimization generally lacks a closed-form solution, making numerical optimization expensive and sensitive to initialization, although variable projection and snapshot alignment can help.Modes may be initialized from leading singular vectors of transformed snapshots after initializing path variables.Source: §5.3.3 · p. 37

5.3.4. Shifted POD: Construction of the reduced model

• When paths are unknown, shifted POD yields an underdetermined differential-algebraic reduced model requiring phase or reconstruction conditions; the coupled model can minimize the residual in a continuous-optimality sense.The reduced system may become singular when coefficients vanish, motivating regularization or shared transformations and path variables.Source: §5.3.4 · p. 38

5.3.5. Shifted POD: Offline–online decomposition

• Offline–online efficiency is generally lost because transformed reduced-model evaluations scale with full-model dimension, except for special unitary single-transformation cases.Sampling and interpolation, transformed empirical interpolation, active subspaces, adaptive time stepping, and machine-learned coefficients or paths provide mitigation strategies.Source: §5.3.5 · p. 39

5.3.6. Shifted POD: Transformation operators

• Transformation design should exploit application expertise and equivariance to preserve high fidelity beyond training data; absent such knowledge, operators can instead be learned from data.Simple shifts suit periodic one-dimensional settings, whereas nonperiodic boundaries and multidimensional waves require extrapolation, padding, or more involved operators.Source: §5.3.6 · p. 40

6. Online adaptive model reduction

• Online adaptive model reduction updates the reduced space over time during the online phase.Source: §6 · p. 40
• Transport-dominated solutions can often be approximated well in reduced spaces that are local in time.Source: §6 · p. 40
• A single global reduced space can be inefficient for transport-dominated problems.Source: §6 · p. 40

Online basis evolution

• Online adaptive methods represent solutions with basis functions that evolve over time, causing the spanned reduced space to evolve as well.In semi-discrete form, the time-dependent basis is represented by an online-updated matrix V(t) ∈ R^N×n, possibly depending on μ.Source: §6.1 · p. 40

Motivation

• Local-in-time low-dimensional spaces can approximate evolving solutions, enabling transport and moving coherent structures without a high-dimensional online weight vector.The reduced space adapts as solutions evolve, unlike a single fixed linear reduced space.Source: §6.1 · pp. 40–41

Computational challenge

• Online basis adaptation weakens offline–online decomposition because updates can scale with the full solution dimension N, motivating mechanisms such as sparse basis updates.These costs can limit speedups even as adaptation improves representational flexibility.Source: §6.1 · p. 41

Scope and benefit

• Online adaptation can capture dynamics and solution features absent from offline training data, especially when high-dimensional parameter domains cannot be sampled exhaustively or training trajectories are scarce.The survey focuses on overcoming the Kolmogorov barrier rather than on avoiding comprehensive offline training, discussed in Peherstorfer and Willcox (2015b) and Remark 6.1.Source: §6.1 · p. 41

Related approaches

• This survey distinguishes adaptation over time to mitigate the Kolmogorov barrier from reduced-space construction over outer-loop iterations, which often addresses unsampled parameter domains.The latter line of work includes Arian, Fahl and Sachs (2000), Kunisch, Karl and Volkwein, Stefan (2008), Zahr and Farhat (2015), Amsallem, Zahr and Washabaugh (2015), and Qian, Grepl, Veroy and Willcox (2017).Source: §6.1 · p. 41

6.2.1. Localized model reduction

• Localized model reduction selects or weights multiple low-dimensional spaces online, targeting parameter, time, state, spatial, or multiscale regimes that a single global space may approximate poorly.Dictionary approaches extend this idea by selecting a few basis functions from a large offline candidate set.Source: §6.2.1 · pp. 41–42

6.2.2. Dynamic low-rank approximations and related methods

• Dynamic low-rank approximations evolve low-rank factors using Dirac–Frenkel projection, but small singular values can cause conditioning and step-size problems requiring specialized integrators.Projector-splitting methods are robust to small singular values, while BUG integrators can be more efficient; stable integration remains an active research direction.Source: §6.2.2 · p. 43
• Related time-dependent methods derive basis evolution from governing equations or dynamic low-rank ideas, including tensor, stochastic, structure-preserving, and precomputed time-dependent reduced-space formulations.These approaches include conservation-law and symplectic structure preservation, orthonormal ADEIM updates, and offline training trajectories used without further online enrichment.Source: §6.2.2 · pp. 43–44, §6.2.3 · p. 45

6.2.3. Time-adaptive basis in model reduction

• Time-adaptive model-reduction bases evolve by mimicking transport, minimizing instantaneous residuals, switching to full-model snapshots, or updating data-driven low-rank spaces online.ADEIM uses sparse residual samples to reduce online costs and lifting bottlenecks, while adaptive sampling is critical for transport-dominated problems and lookahead improves robustness.Source: §6.2.3 · pp. 44–45

6.3.1. Time-discrete full models

• The static baseline uses a fixed basis and interpolation points in a fully discrete reduced model obtained through empirical interpolation, with each step generally requiring a nonlinear solve.ADEIM instead makes the reduced right-hand side step-dependent as the basis and interpolation points evolve.Source: §6.3.1 · p. 46, §6.3.3 · p. 48

6.3.2. ADEIM: Basis and points adaptation

• ADEIM adapts the reduced basis and interpolation points online through low-rank updates, typically with small rank r ≪ n or rank one.This realizes a time-adaptive nonlinear parametrization whose online weights include the reduced state and update variables, while offline weights include the initial basis and interpolation points.Source: §6.3.2 · pp. 46–47
• Basis updates approximate a sliding window of surrogate residual information, while sparse residual minimization avoids the computational cost of a full nonlinear online optimization.The update can be computed from a thin SVD of an ns × w sampled residual matrix, with cost O(ns w^2) when w ≤ ns.Source: §6.3.2 · pp. 47–48

6.3.3. ADEIM: Forward step of reduced model

• At each forward step, ADEIM rebuilds the empirically interpolated reduced right-hand side with the adapted basis and points, then solves the generally nonlinear reduced system.The previous reduced solution is projected onto the new basis before advancing one time step.Source: §6.3.3 · p. 48

6.3.4. ADEIM: Sliding window

• Sliding-window surrogates are formed by evaluating the full-model right-hand side at the lifted reduced solution, revealing basis-missing directions without integrating the full model.Only ns sampling components are evaluated, while unobserved components are approximated by least-squares empirical regression; the resulting cost is comparable to a full-model residual evaluation.Source: §6.3.4 · p. 49

6.3.5. ADEIM: Sampling

• For transport-dominated problems, adaptive sampling selects the ns largest row norms of the residual, and this objective is at most twice the optimal selection objective.Uniform sampling is inefficient for localized features such as wave fronts; selecting points requires full residual evaluation and sorting row norms.Source: §6.3.5 · pp. 50–51

6.3.6. Online algorithm for ADEIM reduced models

• The generic online algorithm initializes from w_init full-model snapshots, then iterates basis adaptation, interpolation-point updating, reduced forward integration, sliding-window updating, and optional sampling adaptation.Basis and sampling updates may be performed intermittently to reduce online costs.Source: §6.3.6 · pp. 51–52
• Online ADEIM time-step costs are typically dominated by evaluating the full-model right-hand side used to update the sliding window.This evaluation determines the computational cost of the online algorithm.Source: §6.3.6 · p. 52
• At sampling-point adaptation steps, evaluating the right-hand side at all N components yields costs scaling at least linearly with N but typically below solving the full nonlinear system (6.1).The all-component evaluation occurs only when sampling points are adapted.Source: §6.3.6 · p. 52
• At other time steps, sparse right-hand-side queries at n_s ≪ N sampling points make the cost scale with n_s rather than N.Thus, most online steps avoid all-component evaluations.Source: §6.3.6 · p. 52

Instantaneous residual minimization methods

• Generic parametrizations allow a time- and parameter-dependent weight vector 𝜽(t, 𝝁) to enter nonlinearly without restricting its structural form.They may also combine with an offline weight vector as in (4.7).Source: §7 · p. 53
• At each time t, instantaneous residual minimization determines the weight-vector derivative 𝜽̇(t, 𝝁) by minimizing the residual norm.The method derives reduced dynamics for generic nonlinear parametrizations through this time-local minimization.Source: §7 · p. 53
• Unlike transformation-based and online adaptive basis methods, generic parametrizations do not prescribe how the weight vector enters or exploit that structure for efficient computation.This distinction motivates the use of instantaneous residual minimization to derive the reduced dynamics.Source: §7 · p. 53

7.2. Instantaneous residual minimization

• The Dirac–Frenkel variational principle provides the core framework for instantaneous residual minimization with generic nonlinear parametrizations.The section also reviews Neural Galerkin schemes built on this principle.Source: §7.1 · p. 53

7.3. Residual approximation

• Residual norms in Neural Galerkin schemes are approximated using static and adaptive collocation methods.These approaches are reviewed as numerical implementations of the variational framework.Source: §7.1 · p. 53

7.4. Time discretization

• The weight-vector evolution induced by residual minimization requires time discretization, whose time-stepping choices and implications are examined.Source: §7.1 · p. 53

7.5. Regularization and solvers

• Regularization addresses cases where instantaneous residual minimization leaves the weight vector underdetermined or produces singular or poorly conditioned systems.The resulting optimality equations are solved at each time step, with randomized sketching used to reduce costs and potentially improve conditioning.Source: §7.1 · p. 53

7.6. Optimization and discretization order

• The section relates the Dirac–Frenkel principle to optimize-then-discretize schemes and presents discretize-then-optimize schemes as an alternative.Source: §7.1 · p. 53

7.7. Trained nonlinear parametrizations

• Instantaneous residual minimization is also considered for snapshot-trained nonlinear parametrizations, including autoencoders and neural representations.Source: §7.1 · p. 53

7.2.1. The Dirac–Frenkel variational principle

• Dirac–Frenkel chooses the weight derivative so the residual is orthogonal to all admissible tangent-space variations of the nonlinear trial manifold.The parametrization must be differentiable in its weights, and its represented functions must belong to the PDE solution space.Source: §7.2.1 · p. 54, Figure 7.1 · p. 57
• The resulting instantaneous conditions act locally in time, unlike global-in-time residual minimization, and may fail to uniquely determine the weight derivative when tangent-space dimension is deficient.The chosen inner product also matters: L2 may not control the error in the relevant norm, while alternative inner products can be difficult to evaluate numerically.Source: §7.2.1 · pp. 54–55

7.2.2. Dirac–Frenkel and instantaneous residual minimization

• Instantaneous residual-norm minimization yields the same conditions as Dirac–Frenkel, while linear parametrizations with orthonormal bases recover the Galerkin reduced model.Thus, Dirac–Frenkel generalizes Galerkin orthogonality and provides a unifying framework for reduced dynamics on nonlinear trial manifolds.Source: §7.2.2 · p. 56

7.2.3. Literature overview of instantaneous residual minimization with nonlinear parametrizations

• The literature applies Dirac–Frenkel to untrained and trained nonlinear parametrizations, including moving finite elements, morphing shapes, matrix factorizations, and neural networks.Neural-network variants address expressivity, geometric domains, stability, high-dimensional sampling, and regularization, while structured parametrizations can enable more efficient specialized solvers.Source: §7.2.3 · pp. 56–57

7.3.2. Estimating inner products with sampling points

• Nonlinear parametrizations require numerical inner-product evaluation because their changing test spaces prevent precomputation and eliminate componentwise residual-norm assembly.Quadrature or collocation over the spatial domain provides the required approximations; closed-form or symbolic evaluation is available only in some settings.Source: §7.3 · pp. 57–58, §7.3.2 · p. 59

7.3.1. Neural Galerkin: An instantiation of the Dirac–Frenkel variational principle

• Neural Galerkin schemes evolve neural-network weights according to Dirac–Frenkel dynamics, equivalently solving a least-squares problem linear in the unknown weight derivative.The derivative is the coefficient vector of the best projection of the PDE right-hand side onto the space spanned by parametrization gradients.Source: §7.3.1 · p. 58

7.3.3. Adaptive sampling points

• Adaptive sampling evolves collocation points with the weights, concentrating them where residuals are large to obtain more efficient residual-norm estimates.Wen et al. (2024) use time-dependent measures, with Gibbs potentials driven by the residual and particle methods such as Langevin sampling or Stein variational gradient descent.Source: Figure 7.2 · p. 60, §7.3.3 · pp. 60–63
• Static collocation can require many points for transport-dominated problems because moving sharp features must remain resolved throughout the spatial domain.Most points may lie in low-information regions while still requiring gradient and right-hand-side evaluations.Source: §7.3.3 · pp. 60–61
• Under standard full-support conditions, the optimal residual is invariant to changing the sampling measure when it can be reduced to zero.The adaptive measure and weight dynamics are coupled, while its adaptation rate is controlled by γ; only samples need to be propagated, not the Fokker–Planck equation explicitly.Source: §7.3.3 · pp. 61–62

Semi-discrete least-squares problem

• Semi-discrete residual minimization uses a batch gradient and batch right-hand side evaluated at m sampled points, which may be static or adaptive.The sample batch can change over time steps and optimization iterations in the time-discrete setting.Source: §7.4 · p. 63
• The sampled matrix formulation represents the empirical objective using the readily evaluated Euclidean vector norm and remains linear in the unknown.Preserving this linearity is a key consideration when discretizing in time.Source: §7.4 · p. 63

Time discretization

• Explicit time discretization requires solving only a linear least-squares problem at each time step.This motivates explicit integration for many Dirac–Frenkel-based methods because it preserves discrete-time linearity.Source: §7.4 · p. 64
• Implicit time discretization makes each time-step problem nonlinear because the batch gradient and right-hand side are evaluated at the next iterate.The resulting nonlinear least-squares fit must be solved at every time step and inherits the difficulties of fitting a nonlinear parametrization such as a neural network.Source: §7.4 · p. 64

7.5.1. Poor conditioning of least-squares problem and tangent space collapse

• Ill-conditioned or rank-deficient batch gradients can make weight updates nonunique, collapse the tangent and test spaces, and cause uncontrolled residual components and low accuracy.Regularization must account for accumulated bias across sequential time steps and may preserve parameter redundancies, preventing recovery from degeneracy.Source: §7.5.1 · p. 65

7.5.2. Regularized Dirac–Frenkel variational principle

• Tikhonov regularization damps small-singular-value directions, whereas truncated SVD discards them; the regularization strength balances stability against bias and residual minimization.Feischl et al. (2024) adapt the strength using the defect, while Lubich and Nick (2025) apply regularized implicit Dirac–Frenkel integration to stiff problems with Gauss–Newton iterations.Source: §7.5.2 · p. 66

7.5.3. Regularization via random subspace approximations

• Right sketching restricts each weight update to an s-dimensional random subspace, reducing the least-squares unknowns from n to s while providing condition-number bounds through the sketch dimension.Independent random updates can be averaged to control variance; Gaussian, random-unitary, and sparse coordinate embeddings are considered.Source: §7.5.3 · p. 67

OtD formulation

• OtD derives continuous-time weight dynamics by enforcing residual orthogonality to the trial-manifold tangent space before discretizing that evolution in time.The Dirac–Frenkel variational principle is interpreted as an OtD approach.Source: §7.6 · p. 67

DtO formulation

• DtO schemes discretize the PDE first, then introduce the nonlinear parametrization and solve a residual-minimization problem for each resulting spatial boundary-value problem.Examples include Kvaal, Lasser, Pedersen and Adamowicz (2023), Chen, Wu, Grinspun, Zheng and Chen (2023a), and Chen, Mccarran, Vizcaino, Soljacic and Luo (2024).Source: §7.6 · pp. 67–68

Linear-model correspondence

• For linear parametrizations, DtO corresponds to the Rothe method, while least-squares Petrov–Galerkin and Galerkin formulations represent DtO and OtD, respectively.The linear-model-reduction correspondence is described using Rothe 1930, Deuflhard and Weiser 2012, and Carlberg et al. (2011, 2017).Source: §7.6 · p. 68

Computational cost

• DtO generally costs more than OtD because each time step entails a typically non-convex optimization, whereas explicit OtD integration yields a linear least-squares problem.The comparison is stated for explicit time integration.Source: §7.6 · p. 68

Conditioning

• DtO can avoid OtD ill-conditioning associated with tangent-space collapse or matrix singularity.This benefit is presented alongside the higher computational cost of DtO schemes.Source: §7.6 · p. 68

7.7. Instantaneous residual minimization with trained nonlinear parametrizations for nonlinear model reduction

• Trained nonlinear parametrizations fix offline weights learned from snapshots, then compute online weights through instantaneous residual minimization.The surveyed methods include neural-network autoencoders, fixed feature maps, and neural representations; transformation-based and online-adaptive methods are treated elsewhere.Source: §7.7 · p. 68

7.7.1. Nonlinear parametrizations given by neural-network autoencoders

• Autoencoder decoders provide nonlinear reduced parametrizations, while residual minimization updates latent states using the decoder Jacobian and full-model right-hand side.Implicit time discretizations require nonlinear least-squares solves at every time step, and constraints can preserve mass, momentum, or energy.Source: §7.7.1 · pp. 68, 70
• Autoencoder residual minimization can lose online speedups because residuals, Jacobians, decoder lifting, and right-hand-side evaluations remain full-dimensional.Masked decoders, sparse collocation, and compressed decoders reduce these costs by restricting evaluations to selected spatial points (Kim et al. 2022; Romor, Stabile and Rozza 2023).Source: §7.7.1 · p. 71

7.7.2. Autoencoders with a priori feature maps and quadratic approximations

• Quadratic feature-map decoders combine a linear approximation with nonlinear corrections, yielding structured trial manifolds whose expressivity depends on the prescribed feature map.For polynomial degree α, Buchfink et al. (2024a) report a linear-advection n-width lower bound behaving roughly as O(n^-α/2).Source: §7.7.2 · pp. 72–73
• Linear encoders can match generic nonlinear encoders in reconstruction while improving training robustness, and nonlinear decoders can yield transported-feature errors decaying faster than the Kolmogorov n-width.These findings are reported by Cohen et al. (2024), Glas and Unger (2025), and Schwerdtner, Gugercin and Peherstorfer (2025a).Source: §7.7.2 · p. 73
• Quadratic structure enables closed-form affine Jacobians, offline–online separation, and sparse residual evaluations that can reduce online costs, including N-independent assembly for linear affine models.Selected collocation points and Jacobian approximations further reduce evaluations in approaches by Sharma et al. (2023), Weder et al. (2025), Barnett and Farhat (2022), and Geelen et al. (2023b).Source: §7.7.2 · p. 75
• A priori fixed feature maps and quadratic approximations provide low-order Jacobian structure but generally offer less expressivity than deep neural networks with representation learning.Generic neural-network parametrizations require online automatic differentiation, potentially within every Gauss–Newton iteration and time step.Source: §7.7.2 · pp. 75–76

7.7.3. Neural representations with offline–online weights

• Offline–online neural representations train fixed weights once while determining few time- and parameter-dependent online weights through modulation, including continuous low-rank adaptation layers.The online weights can be generated by a modulation network h_ψ during training to encourage temporal regularity, while full-model solution functions can be used directly across meshes and discretizations.Source: §7.7.3 · pp. 76–78
• Neural representations combine coordinate-to-value parametrizations with instantaneous residual minimization using sampling points independent of the full-model dimension, avoiding the classical lifting bottleneck.Berman and Peherstorfer (2024) leverage Neural Galerkin schemes, while constraints and nonlinear embedding steps can enforce mass, momentum, and energy conservation.Source: §7.7.3 · p. 78
• Online costs are governed mainly by the number of sampling points, batch-gradient computation, and full-model right-hand-side evaluations, with spatial derivatives obtained through automatic differentiation.Explicit time integration requires one linear least-squares problem per time step with n unknowns; accuracy requires sufficiently many informative sampling points, which may need to reflect fine full-model resolution.Source: §7.7.3 · pp. 78–79

• Generic nonlinear parameterizations combined with instantaneous residual minimization provide a unifying lens, while transformation-based and online adaptive methods expose additional structural features for efficiency and robustness.The survey reports significant progress and a rapidly growing literature on transport-dominated model reduction.Source: §8 · p. 79
• Important underrepresented directions include structure-preserving reduction, stochastic systems, control applications, and hyperbolic PDE reduction through delay-equation representations, with early developments by Schulze, Unger, Beattie and Gugercin (2018).Source: §8 · p. 79
• Non-intrusive model reduction learns low-dimensional representations and reduced dynamics primarily from snapshot data, unlike intrusive derivations from governing equations and unlike equation discovery.The outlook points to Ghattas and Willcox (2021) and Kramer et al. (2024) for overviews.Source: §8 · pp. 79–80
• Non-intrusive methods address transport-dominated regimes through nonlinear transformations that factor out transport or learned nonlinear latent representations with separately learned reduced dynamics.Examples include transformation-based approaches by Sesterhenn and Shahirpour (2019), Sarna and Benner (2021), Lu and Tartakovsky (2020), Issan and Kramer (2023), and Gowrachari, Demo, Stabile and Rozza (2025), alongside autoencoder-based approaches such as Fresca, Dede’ and Manzoni (2021) and Maulik et al. (2021).Source: §8 · p. 80