PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations

Joan Bruna, Benjamin Peherstorfer, Eric Vanden-Eijnden

arXiv:2203.01360v4math.NAcs.LGcs.NAstat.ML

TL;DR

High-dimensional PDEs and localized, evolving solution features make informative training-data acquisition challenging for neural solvers. Neural Galerkin schemes use Dirac-Frenkel parameter dynamics with adaptive sampling guided by the PDE and current solution. Experiments show efficient capture of solution evolution in high-dimensional transport and interacting-particle cases, while the paper identifies unresolved approximation and estimation questions.

  • Problem

    In high-dimensional domains with localized solution structures, uninformed sampling can yield inaccurate residual estimates even when the neural representation is expressive.

    Source: §2.2 · p. 4

  • Method

    Neural Galerkin schemes evolve network parameters sequentially via the Dirac-Frenkel principle and adapt residual sampling using the PDE dynamics and current solution.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    The method efficiently captures solution evolution in variable-speed hyperbolic advection and interacting-particle Fokker-Planck cases, overcoming collocation sampling limitations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The schemes intertwine adaptive data acquisition and simulation, using solution-informed sampling to learn PDE solutions over time.

    Source: §6 · p. 20

    Uniform-in-time approximation guarantees and conditions for efficient instantaneous-loss estimation remain open questions, and kernel estimators may not work well for spatially localized structures.

    Source: §4.1 · p. 7, §6 · p. 21

Abstract

from arXiv · show

Deep neural networks have been shown to provide accurate function approximations in high dimensions. However, fitting network parameters requires informative training data that are often challenging to collect in science and engineering applications. This work proposes Neural Galerkin schemes based on deep learning that generate training data with active learning for numerically solving high-dimensional partial differential equations. Neural Galerkin schemes build on the Dirac-Frenkel variational principle to train networks by minimizing the residual sequentially over time, which enables adaptively collecting new training data in a self-informed manner that is guided by the dynamics described by the partial differential equations. This is in contrast to other machine learning methods that aim to fit network parameters globally in time without taking into account training data acquisition. Our finding is that the active form of gathering training data of the proposed Neural Galerkin schemes is key for numerically realizing the expressive power of networks in high dimensions. Numerical experiments demonstrate that Neural Galerkin schemes have the potential to enable simulating phenomena and processes with many variables for which traditional and other deep-learning-based solvers fail, especially when features of the solutions evolve locally such as in high-dimensional wave propagation problems and interacting particle systems described by Fokker-Planck and kinetic equations.

1. Introduction

High-dimensional PDEs challenge grid-based solvers, while global collocation can struggle to sample localized, evolving solution features. Neural Galerkin schemes combine sequential parameter updates with adaptive sampling, and experiments demonstrate their use on high-dimensional transport and particle systems.

1. Introduction

  • Grid-based methods face exponentially increasing computational costs with dimension, limiting classical PDE solvers for systems such as interacting-particle equations.The introduction names Schrödinger, Fokker-Planck, and Boltzmann equations as examples.Source: §1 · p. 1
  • Global time-space collocation usually samples without tailoring point locations to the unknown solution, which can impair high-dimensional adaptation and long-range initial-value prediction.The passage also notes that treating space and time equivalently can lead to a loss of causality.Source: §1 · p. 2
  • Neural Galerkin schemes combine Dirac-Frenkel parameter updates over time with adaptive sampling guided by the evolving solution and PDE dynamics.The sampling measure adapts to follow important solution features, without requiring a priori solution data.Source: §1 · p. 2
  • The scheme derives nonlinear evolution equations for representation parameters, which standard solvers can integrate with time steps adapted to solution dynamics.Suitable integrators can take larger steps when possible and reduce step sizes when the dynamics require it.Source: §1 · p. 2
  • Dynamical loss estimation with adaptive sampling is described as essential for capturing solution details that static sampling misses and for interpreting complicated solutions through feature statistics.Source: §1 · p. 2
  • Experiments show efficient solution capture for variable-speed hyperbolic advection and interacting-particle Fokker-Planck equations, overcoming sampling limitations of collocation methods.The introduction also describes KdV solitons and Allen-Cahn domain walls as low-dimensional examples of moving localized features.Source: §1 · pp. 2–3
  • Prior collocation-based and task-specific approaches have limited adaptive sampling or scope, while many surrogate methods require unavailable ground-truth PDE solutions.Neural Galerkin schemes instead integrate PDE-solution parameters directly, avoiding the costly lifting required by one embedding-based approach.Source: §1 · p. 3
  • Earlier Dirac-Frenkel approaches focused on parametrizations or PDEs with analytically computable objectives, or on low-dimensional problems where static uniform sampling is tractable.The proposed adaptive sampling approach is aimed at high-dimensional problems with generic nonlinear parametrizations.Source: §1 · p. 3

2. Setup and problem formulation

The section formulates time-dependent PDE solution as an evolution problem and contrasts global time-space fitting with sequential solution modeling. It identifies Monte Carlo estimation under uninformed sampling as a challenge when high-dimensional solutions develop localized structures.

2. Setup and problem formulation

  • The target problems are time-dependent PDEs on potentially high-dimensional spatial domains whose solutions can contain local features transported over time.Source: §2 · p. 3

2.1. Evolution equations

  • The evolution equation specifies the time derivative of the field through a function f of time, space, and the field value, with an initial condition u0.Source: §2.1 · pp. 3–4
  • Choosing f appropriately yields different PDEs, including an advection-diffusion-reaction equation.Source: §2.1 · p. 4
  • The formulation assumes boundary conditions that make the evolution equation well-posed, so its solution exists uniquely and depends continuously on u0.Source: §2.1 · p. 4

2.2. Problem formulation

  • Typical collocation methods represent the solution with a neural network that takes both time and space as inputs and globally minimizes expected squared residual over time-space.Source: §2.2 · p. 4
  • Monte Carlo estimation of the expected residual draws time-space samples from a distribution and minimizes the empirical average of squared residuals.Source: §2.2 · p. 4
  • In high dimensions, uninformed sampling can poorly estimate residuals when solutions develop localized structures, with estimator variance undermining optimization despite expressive network representations.Source: §2.2 · p. 4

3. Neural Galerkin schemes via the Dirac-Frenkel variational principle

The Neural Galerkin approach represents a time-evolving PDE solution through parameters governed by the Dirac-Frenkel variational principle. It minimizes a time-local residual to derive parameter dynamics, then numerically integrates those dynamics using standard time integrators.

3. Neural Galerkin schemes via the Dirac-Frenkel variational principle

  • The Dirac-Frenkel formulation carries PDE dynamics into network parameters, enabling information from earlier time steps to inform the solution and residual-sampling measure at later times.Source: §3 · p. 4

3.1. Nonlinear parametrizations that depend on time

  • The solution ansatz places time in the evolving parameter θ(t), rather than as a network input, and permits nonlinear dependence on those parameters.Finite-dimensional representations such as DNNs introduce approximation errors that the method aims to keep low over time.Source: §3.1 · p. 5

3.2. Training nonlinear parametrization via the Dirac-Frenkel variational principle

  • The method minimizes the residual at each time to obtain an initial-value problem, rather than a global-in-time boundary-value problem requiring storage of the full parameter path.The instantaneous residual depends on the current parameter and its time derivative.Source: §3.2 · p. 5

3.3. System of ODEs for network parameters

  • The resulting parameter ODEs define the Neural Galerkin solution through residual testing with the parameter-gradient functions in the L2(X; ν) inner product.Source: §3.3 · p. 6

3.4. Discretization in time

  • The parameter ODEs can be numerically integrated with standard explicit or implicit methods using possibly nonuniform, adaptive time steps.Source: §3.4 · p. 6

3.4.1. Explicit integrators.

  • Forward Euler gives an explicit update, while RK45 can also be used and leads to linear systems at each stage.A regularizing term λ Id may be added to the matrix in the explicit equation.Source: §3.4.1 · p. 6

3.4.2. Implicit integrators.

  • Implicit integrators cost more per step because they require solving nonlinear equations; the authors also note a discretize-then-derive alternative to continuous-time formulation.The adaptive approach differs from time-space collocation and methods formulated over equidistant time steps.Source: §3.4.2 · p. 7

4. Active learning with Neural Galerkin schemes

Neural Galerkin active learning adapts the sampling measure as the network parameters evolve, aiming to estimate the operators M and F accurately. Sampling choices can be matched to solution structure and network architecture.

4.1. Active learning via time-dependent measures

  • High-dimensional quadrature becomes intractable, while uniform Monte Carlo may miss spatially localized solution structures; a parameter-dependent measure can lower estimator mean-squared error.The measure νθ depends on the evolving parameters, allowing it to adapt over time.Source: §4.1 · p. 7

4.2. Monte Carlo estimators with adaptive samples

  • Adaptive Monte Carlo samples must reflect the current solution; for localized probability densities, choosing νθ = U(θ) can yield low-variance estimates.With Gaussian units, integrals for components of M can be evaluated analytically, leaving sampling to estimate F; gradient-based measures are another option for localized fronts.Source: §4.2 · p. 8

4.3. Importance sampling with respect to a nominal measure

  • Importance sampling constructs νθ from a nominal measure using positive parameter-dependent weights, then uses reweighted estimators for M and F.Source: §4.3 · p. 8

4.4. Neural architectures and sampling

  • The network architecture can be chosen to align with adaptive sampling; shallow networks with Gaussian units allow direct sampling from U(θ) when it represents a probability density.The network’s coefficients and features are its parameters.Source: §4.4 · pp. 8–9

5. Numerical experiments

The experiments compare Neural Galerkin approximations with benchmarks and fixed-feature linear Galerkin methods, including high-dimensional advection cases where adaptive sampling is tested against static sampling. Across the examples, evolving network features and sampling adapted to the solution support accurate approximations of local dynamics.

5. Numerical experiments

  • The experiments span low-dimensional KdV and Allen-Cahn benchmarks and moderately high-dimensional PDEs with local solution features.The numerical examples include high-dimensional advection.Source: §5 · p. 9, §5.4 · p. 13

5.1. Network architectures

  • The experiments use shallow networks with Gaussian or periodic sine units and feedforward networks with multiple hidden layers and tanh units.For the feedforward architecture, the parameters include output coefficients, weight matrices, and biases.Source: §5.1 · p. 9

5.2. Korteweg-de Vries equation

  • For KdV, Neural Galerkin remained close to the analytic solution, whereas fixed-feature linear Galerkin errors became orders of magnitude larger despite matching degrees of freedom.The linear approximation developed large errors after only a few time steps.Source: §5.2 · pp. 9, 11, Figure 1 · p. 10

5.3. Allen-Cahn (AC) equation

  • The Allen-Cahn Neural Galerkin approximation achieved an ℓ2 error two orders of magnitude lower than linear Galerkin with the same 16 degrees of freedom.The three-hidden-layer network also captured the sharp-wall dynamics in the reported experiment.Source: §5.3 · pp. 11–12, Figure 3 · p. 12
  • In Allen-Cahn, two nonlinear output components adapt their shapes over time to approximate the solution, despite the network using only two components.The plots show the components, their coefficients, and the benchmark solution.Source: §5.3 · p. 12, Figure 4 · p. 13

5.4.1. Advection with a time-dependent coefficient.

  • In the time-varying advection experiment (κ = 1), uniform sampling over [0,15]^d produced 100% relative error after a few time steps, even with n = 105 points; adapted sampling yielded small ℓ2 error.Uniform sampling covered the solution-propagation domain for t ∈ [0,2].Source: Figure 5 · p. 14, §5.4.1 · p. 16

5.4.2. Advection with time- and space-dependent coefficients.

  • Forward-then-backward integration for time- and space-varying advection returned a solution close to the initial condition, while static uniform sampling failed after a few steps.The analytic solution was unavailable in closed form, so error was assessed by comparing the final and initial solutions.Source: Figure 6 · p. 15, §5.4.2 · p. 17

5.5. Interacting particle systems

The section applies Neural Galerkin schemes to interacting-particle Fokker–Planck equations, approximating the joint density directly with adaptive sampling. In harmonic and aharmonic traps, adaptive sampling tracks particle statistics better than static sampling and supports entropy estimates.

5.5. Interacting particle systems

  • The scheme approximates the joint particle density directly, enabling quantities such as entropy that Monte Carlo samples alone do not directly provide.Sampling to estimate M and F is adaptive and based on the current approximation.Source: §5.5 · p. 17

5.5.1. Particles in harmonic trap.

  • For the harmonic trap, the adaptive entropy estimate closely matches the benchmark, whereas static sampling and Monte Carlo with density estimation from 5,000 samples give poor approximations.The Neural Galerkin approximation provides a density, unlike Monte Carlo's samples alone.Source: §5.5.1 · p. 18, Figure 9 · p. 19
  • In the harmonic trap, adaptive Neural Galerkin predicts the mean with roughly 10^-4 relative error and covariance entries with about 10^-1 to 10^-2 relative error; uniform sampling yields poor approximation.Source: §5.5.1 · p. 18, Figure 8 · p. 19

5.5.2. Particles in aharmonic trap.

  • In the aharmonic trap, adaptive sampling achieves relative errors between 10^-3 and 10^-2 for the mean and covariance diagonal, while static sampling has large errors.Off-diagonal covariance entries converge to zero, so their relative errors are not informative and are not plotted; the adaptive entropy estimate agrees with Monte Carlo and density estimation from 100,000 samples.Source: §5.5.2 · p. 20

6. Concluding remarks

The concluding remarks emphasize adaptive function approximation and data collection as key to accurately predicting local features in high-dimensional domains. They propose broader applications while identifying guarantees, efficient loss estimation, and adaptive sampling strategies as future research questions.

6. Concluding remarks

  • Adaptivity in both function approximation and data collection is key to accurate prediction of local features in high-dimensional domains.The schemes intertwine sampling and numerical simulation, using importance sampling informed by the current solution.Source: §6 · p. 20
  • The authors suggest applying Neural Galerkin schemes to kinetic, nonlinear Fokker–Planck, and Boltzmann equations, especially where solutions lack a Feynman–Kac representation.These are proposed applications, not results established in the conclusion.Source: §6 · pp. 20–21
  • Future work includes understanding architecture conditions for uniform-in-time approximation, efficient loss estimation, and adaptive strategies that maintain a changing neuron budget.The paper also points to importance sampling adapted to transient regularity structures as a possible direction.Source: §6 · p. 21

A. Learning in the Banach space F1

This appendix considers Neural Galerkin representations with an infinite-dimensional parameter given by a measure. It describes the associated function space and its connection to shallow networks.

A. Learning in the Banach space F1

  • An infinite-dimensional Neural Galerkin representation uses a Radon measure with finite total variation over the nonlinear-unit parameter space.The resulting functions form a Banach space whose norm is the minimum total variation among measures representing the same function; shallow two-layer networks approximate them in the infinite-width limit.Source: A. Learning in the Banach space F1 · p. 24
  • The appendix introduces an evolution equation for the measure through the Euler–Lagrange equation associated with a minimization problem.The passage identifies the role of the Euler–Lagrange equation but does not specify the evolution equation in the supplied text.Source: A. Learning in the Banach space F1 · p. 24

B. Another derivation of the Neural Galerkin equation M(θ) ˙θ = F(t, θ)

The appendix derives Neural Galerkin parameter dynamics from an objective for tracking the PDE solution. Under exact representation and stated conditions on the objective, minimizing its instantaneous form recovers dynamics structurally similar to the main-text equation.

B. Another derivation of the Neural Galerkin equation M(θ) ˙θ = F(t, θ)

  • When the current solution is exactly representable and the objective satisfies the stated conditions, its value is zero and can be kept zero or as close to zero as possible as the solution evolves.The proposition derives an instantaneous objective for parameter velocity under these assumptions.Source: B. Another derivation of the Neural… · p. 25
  • Minimizing the instantaneous objective over parameter velocity yields an evolution equation structurally similar to the Neural Galerkin equation in the main text.The objective is derived using the assumed PDE evolution and conditions on R.Source: B. Another derivation of the Neural… · p. 25
  • The objective formulation is more general because it provides a principled way to specify the measure ν_θ, including a Kullback–Leibler objective for probability densities.The supplied text relates a particular measure choice to the main-text parameter equations.Source: B. Another derivation of the Neural… · p. 25

C. Comparison with global methods

The comparison formulates a global-in-time parameter trajectory using the spatial and temporal variables together, as in Deep Ritz [44] [4]. Its Euler–Lagrange equations form a boundary value problem that is harder to solve than the Neural Galerkin equation.

C. Comparison with global methods

  • The global-in-time comparison treats spatial and temporal variables on the same footing, as in Deep Ritz [44] [4].The parameter trajectory is described in the L2([0, T]×X, ν0) sense and is subject to θ(0) = θ0.Source: C. Comparison with global methods · p. 26
  • The Euler–Lagrange equations impose an initial condition and a terminal condition on the parameter trajectory.The displayed form includes θ(0) = θ0 and ∂ηJ_T(θ(T), ˙θ(T)) = 0.Source: C. Comparison with global methods · p. 26
  • The resulting equations are a boundary value problem, which is harder to solve than the Neural Galerkin equation (11).Source: C. Comparison with global methods · p. 26

D. Advection in unbounded, high-dimensional domains

The advection examples compare time-varying and time-and-space-varying transport, showing how spatial variation changes the propagated structure. Forward and backward integration yields matching initial- and end-time approximations in the time-and-space-varying case.

D. Advection in unbounded, high-dimensional domains

  • All five marginals are shown for both the time-varying and time-and-space-varying cases.Figures 11 and 12 present the respective cases.Source: D. Advection in unbounded,… · p. 26
  • With spatially varying coefficients, the propagated coherent structure changes shape, unlike in the time-only-varying case.Source: Figure 12 · p. 27
  • Forward and backward integration produces initial- and end-time approximations that match well, indicating approximation of the local high-dimensional dynamics.This comparison is shown in the second row of Figure 12.Source: Figure 12 · p. 27

E. Fokker-Planck equation in high-dimensions

The particle examples evaluate Neural Galerkin predictions against ODE and Monte Carlo benchmarks for harmonic and anharmonic traps. The figures visualize particle positions over time.

E. Fokker-Planck equation in high-dimensions

Related papers

Loading 2203.01360v4…