PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-based Deep Learning

N. Thuerey, B. Holzschuh, P. Holl, G. Kohl, M. Lino, Q. Liu, P. Schnell, F. Trost

arXiv:2109.05237v4cs.LGphysics.comp-ph

TL;DR

This guide addresses how deep learning can be combined with physical models for simulation, including the challenge that supervised learning can average distinct valid solutions. It presents approaches ranging from residual constraints and differentiable simulations to generative methods, and describes improved solution quality with differentiable solvers and a probabilistic airfoil surrogate that outperforms traditional simulators.

  • Problem

    Supervised training on multiple valid solution modes can learn an undesirable mean unlike any individual solution.

    Source: §46.5 · p. 449

  • Method

    The guide presents physics-based deep-learning approaches with varying degrees of physical-model integration, including equation residuals, differentiable simulations, and generative methods.

    Source: Coming up · p. 11, viii · p. 11

  • Results

    When problem knowledge is available, differentiable physics can significantly improve training and solution quality, and diffusion modeling provides an airfoil-flow surrogate that outperforms traditional simulators.

    Source: Coming up · p. 11, §15.5 · p. 145

  • Takeaways & Limitations

    Which physical-model integration approach is useful depends on the scenario.

    Source: Coming up · p. 11

    SIP’s use of first-order optimizers for Δθ can perform poorly in ill-conditioned settings, especially when network outputs have very different scales.

    Source: §38.5.1 · p. 403

Abstract

from arXiv · show

This document is a hands-on, comprehensive guide to deep learning in the realm of physical simulations. Rather than just theory, we emphasize practical application: every concept is paired with interactive Jupyter notebooks to get you up and running quickly. Beyond traditional supervised learning, we dive into physical loss-constraints, differentiable simulations, diffusion-based approaches for probabilistic generative AI, as well as reinforcement learning and advanced neural network architectures. These foundations are paving the way for the next generation of scientific foundation models. We are living in an era of rapid transformation. These methods have the potential to redefine what's possible in computational science.

viii

This section is a hands-on guide to physics-based deep learning, pairing concepts with executable Jupyter notebooks and physical models. It spans probabilistic generation, PDE notation and fluid simulations, differentiable physics, neural architectures, reinforcement learning, and optimization methods from Newton’s method to gradient descent and Adam.

viii

  • The v0.3 edition adds a generative-modeling chapter and neural architectures designed for physics simulations, and updates code examples to use the latest frameworks.The new chapter covers denoising, flow matching, autoregressive learning, physics-integrated constraints, and diffusion-based graph networks.Source: viii · p. 11

Coming up

  • The guide previews airfoil-flow diffusion surrogates, equation-residual training, differentiable simulations, simulator-in-the-loop inverse problems, and architecture choices.It presents physics-based deep-learning approaches in order of increasing integration with physical models and discusses their pros, cons, and useful scenarios.Source: Coming up · p. 11
  • The book uses executable Jupyter notebooks so readers can run and modify code examples directly in a browser.The text also presents notebooks as a form of literate programming.Source: Coming up · p. 12

A TEASER EXAMPLE

  • Flow matching captures both solution modes of the parabola, while supervised learning averages them and differentiable physics represents one mode.In the teaser, conventional supervised learning produces an approximation away from both modes; a discretized solver can represent one mode, while flow matching learns the full distribution.Source: A TEASER EXAMPLE · p. 17, Figure 1.1 · p. 17

1.1 Differentiable physics

  • Differentiable physics integrates discretized physical models into neural-network training, requiring differentiable operators to provide learning gradients.The guide contrasts this with supervised training on precomputed data and describes model equations such as time evolution or conservation laws.Source: §1.1 · pp. 17–18

1.2 Finding the inverse function of a parabola

  • Supervised training on both inverse-parabola modes averages their solutions and fails to find either valid mode.The example randomly samples positive and negative square-root solutions; its red prediction lies away from both modes.Source: §1.2 · pp. 18, 20

1.3 A differentiable physics approach

  • Using the parabola operator in the loss learns a good inverse without averaging the two modes, but the deterministic network represents only one side.The selected side depends on random initialization, and the result needs further attention near the origin.Source: §1.3 · pp. 21–23

A PROBABILISTIC GENERATIVE AI APPROACH

  • The probabilistic approach uses flow matching to learn the full distribution of possible inverse solutions, with noisy data helping train the network toward both modes.The section introduces a dataloader that adds different amounts of noise to the solution values.Source: A PROBABILISTIC GENERATIVE AI APPROACH · p. 25
  • The flow-based probabilistic approach recovers both solution modes above and below the x-axis, though its output remains noisy.Flow integration transforms an initial random distribution toward the bimodal parabola targets; the text suggests a larger network could reduce the noise. The approach is trained on data alone, while physics priors may help drive noisy outputs toward accurate solutions.Source: A PROBABILISTIC GENERATIVE AI APPROACH · pp. 27–29

2.1 Discussion

  • Supervised learning can average distinct modes into an unphysical solution, making recovery of modes important for practical PDEs such as flow bifurcations.The text motivates differentiable-physics training for more complex cases.Source: §2.1 · p. 29, Figure 2.1 · p. 29

2.2 Next steps

  • The notebooks are executable starting points for experiments, with intentionally small datasets and networks; suggested extensions include changing training or architectures and solving a 2D paraboloid inverse problem.The book does not cover every type of physical model or simulation.Source: §2.2 · p. 30, §3.3 · p. 34

3.1.1 Replacing traditional simulations?

  • Deep learning methods are presented as numerical tools, not replacements for simulation; the text emphasizes combining them with domain knowledge and established methods.It describes deep learning as a powerful toolkit for nonlinear problems while acknowledging that integrating physical understanding into learning algorithms can be non-trivial.Source: §3.1.1 · p. 32, §3.1.2 · p. 32

3.1.3 Reconciling AI and simulations

  • The book aims to reconcile AI with physical simulation by using deep learning for PDEs, incorporating physics knowledge, and retaining numerical methods.It argues that specialized discretization methods should be used where possible, since seemingly small changes can determine whether key phenomena appear in solutions.Source: §3.1.3 · pp. 32–33, §3.1.1 · p. 32
  • For repeatedly solved domains, trained neural networks could support numerical solvers, and domain-specific foundation models could enable hybrid solvers to outperform generic ones.The proposed approach fine-tunes a model with a smaller dataset for a domain-specific specialization.Source: §3.1.3 · p. 33

3.2 Categorization

  • Physics-based deep learning spans forward and inverse problems and ranges from supervised data, through physics-informed loss terms, to hybrid methods interleaving simulations with neural networks.Hybrid approaches provide the tightest coupling between physical models and learning.Source: §3.2 · pp. 33–34

3.2.1 Naming

  • “Differentiable physics” is equivalent to coupling an adjoint method with deep learning or backpropagating through a numerical simulation; hybrid networks can correct numerical errors or unresolved equation terms.Related formulations include closure in fluid dynamics, homogenization or coarse-graining in materials, and parametrization in climate and weather simulation.Source: §3.2.1 · p. 34

3.3 Looking ahead

  • The guide leaves dense field-based simulations, other combinations of machine learning with deep learning, and replacing synthetic data with real-world observations as future directions.Experiments using real-world observations are explicitly left as an outlook.Source: §3.3 · p. 34

3.4 Implementations

  • The guide primarily uses PyTorch, also introduces TensorFlow and PhiFlow, and gives JAX examples; stochastic notebook optimizations can produce different results across runs.Consequently, values discussed in the text may not exactly match values obtained by rerunning the examples.Source: §3.4 · p. 35

3.5.1 Deep learning and neural networks

  • Deep learning approximates ground-truth functions with neural networks by minimizing a scalar loss, typically using stochastic gradient descent and autodiff for weight gradients.Training, validation, and test data are distinguished; test data should assess generalization to an out-of-distribution input distribution.Source: §3.5.1 · p. 35

3.5.2 Partial differential equations as physical models

  • The physical models are continuous PDE operators whose solutions require initial and boundary conditions, and numerical discretization introduces spatial and temporal errors.The text assumes first and second derivatives exist and uses non-dimensional parameterizations that can be rescaled to real-world quantities.Source: §3.5.2 · p. 36

3.5.3 Some example PDEs

  • Burgers’ equation combines advection and diffusion and can form shocks, while Navier-Stokes adds mass conservation and a divergence-free motion constraint.The text also introduces a Boussinesq variant for varying density, including temperature changes.Source: Burgers · p. 37, Navier-Stokes · p. 37

3.6 Simple Forward Simulation of Burgers Equation with phiflow

  • PhiFlow provides differentiable simulation building blocks that interface with deep-learning frameworks, and the forward-simulation example combines explicit diffusion with semi-Lagrangian advection.The example sets up 128 cells in a periodic 1D domain and advances the state through 32 time steps.Source: §3.6 · p. 38, §3.6.2 · p. 39, §3.6.3 · p. 40

3.6.4 Visualization

  • The 1D Burgers simulation develops a central shock as the opposing initial velocity bumps collide.The initial state is −sin(πx); the example uses a periodic domain.Source: §3.6.2 · p. 39, §3.6.4 · p. 41

3.6.5 Next steps

  • The Burgers simulation is a simple starting point for comparing physics-based learning approaches; suggested extensions include changing parameters or initialization and moving to higher dimensions.For the Navier–Stokes example, suggested variations include spatial resolution, buoyancy factors, and simulation duration.Source: §3.6 · p. 42, §3.6.5 · p. 42, §3.7.5 · p. 49

3.7 Navier-Stokes Forward Simulation

  • The example models incompressible flow with a temperature marker that drives buoyancy, on a closed domain with velocity and pressure boundary conditions.It uses a Boussinesq approximation rather than explicitly computing density and discretizes the 100 × 80-unit domain with 40 × 32 cells.Source: §3.7 · p. 43, §3.7.2 · p. 43

3.7.3 Datatypes and dimensions

  • PhiFlow Grid shapes record batch, spatial, and channel dimensions with names and types; here smoke and inflow are 32 × 40, and velocity is 32 × 40 with vector channel x,y.Source: §3.7.3 · pp. 45–46

3.7.4 Time evolution

  • 0.4630011022090912 at frame 0 and 5.483874320983887 at frame 9 were the reported maximum velocities as the hot plume began to rise.The inflow is off-center to the left, so the plume is expected to curve right when it reaches the top wall.Source: §3.7.4 · pp. 46–47
  • The Navier–Stokes simulation is compatible with deep-learning backpropagation and can be extended to more complex cases or 3D.Its update advects smoke and velocity, applies Boussinesq buoyancy, and uses a pressure solve to make velocity divergence free.Source: §3.7.2 · p. 45, §3.7.4 · p. 48

3.8 Optimization and Convergence

  • The optimization chapter connects classical algorithms, beginning with Newton’s method, to deep-learning optimizers and highlights their relevance to physics-based learning.It is presented as an optional theoretical deep dive that skips evaluations and code, with practical readers directed to supervised training.Source: §3.8 · p. 49

3.8.1 Notation

  • The chapter defines notation for a scalar loss, its optimum, update steps, and derivatives, and reviews Taylor expansions, the fundamental theorem of calculus, Lipschitz continuity, and Cauchy–Schwarz.The gradient is the transpose of the loss Jacobian, and the Hessian is its second derivative.Source: §3.8.1 · p. 49, §3.8.2 · p. 50

3.8.3 Newton’s method

  • Newton’s method approximates the loss near a minimum by a parabola and uses local derivative information to compute an update toward the optimum.The derivation’s convergence discussion assumes the Jacobian is differentiable.Source: §3.8.3 · pp. 50–51
  • Newton’s method converges quadratically near the optimum, but its convergence is not guaranteed because the pre-factor K 2H can exceed 1.The passage says the quadratic-convergence expression is exact, with no truncation.Source: §3.8.3 · p. 51

3.8.4 Adaptive step size

  • With a freely chosen adaptive step size, Newton’s method decreases loss for sufficiently small steps and is provably convergent, but requires the Hessian that is difficult to obtain.The proof assumes a symmetric positive-definite Hessian with Lipschitz continuity; it does not apply to a fixed step size.Source: §3.8.4 · pp. 52–53

3.8.6 Broyden’s method

  • Broyden initializes the Hessian approximation with the identity and updates it by finite differences, but assumes the current Jacobian is zero and therefore gives a rough approximation.Its update avoids computing the full Hessian and can be evaluated efficiently.Source: §3.8.6 · pp. 53–54

3.8.7 BFGS

  • BFGS avoids assuming a zero Jacobian and subtracts redundant update components, while practical variants use line search and reduced Hessian representations to save memory.The passage identifies BFGS variants as the most widely used algorithms for classical nonlinear optimization.Source: §3.8.7 · p. 54

3.8.8 Gauss-Newton

  • Gauss-Newton approximates the Hessian with the squared Jacobian for an L2 loss, yielding an approximate inverse-Jacobian step, but large-matrix inversion can be costly or impossible.The resulting step approximately equalizes step sizes across parameters.Source: §3.8.8 · pp. 54–55

3.8.10 Adam

  • Adam replaces Gauss-Newton’s costly matrix inversion with a diagonal Hessian approximation, using momentum for stabilization while neglecting correlations between weights.Its square-root normalization produces approximately fixed-size steps across dimensions; momentum averages gradients and squared gradients to make estimates more robust.Source: §3.8.10 · p. 55

3.8.11 Gradient Descent

  • Gradient descent assumes H = 1, so its update uses the gradient without inversion, which the text notes leaves it in the wrong space for physics problems.Under convexity, differentiability, and a Lipschitz gradient, choosing λ ≤ 1/ℒ ensures convergence; ℒ is generally unknown in deep-learning practice, and the discussion excludes stochastic methods.Source: §3.8.11 · pp. 55–56

Neural Surrogates and Operators

The section presents supervised learning as a starting point for physics-based surrogates, then surveys architecture choices shaped by data structure and local or global dependencies. It compares how convolutional, spectral, and attention-based methods represent those dependencies and the tradeoffs involved.

SUPERVISED TRAINING

  • Supervised learning fits a neural network to input-output data, often generated by simulations; its usefulness depends on model fidelity and numerical accuracy.Trained networks can be cheap to evaluate, although intermediate activations can require substantial memory.Source: SUPERVISED TRAINING · p. 59, §4.1 · p. 59, §4.2 · p. 60

5.1 Spatial Arrangement

  • Spatial arrangements range from none to regular grids, unstructured meshes, and unconnected points; without spatial information, only dense networks apply.Neural fields supplied with positions likewise lack architectural neighbor queries; discretization can introduce more structure.Source: §5.1 · p. 61, §5.2 · p. 62

5.3 Local vs Global

  • Architecture choice should match the data’s dependencies: local-only receptive fields struggle with global effects, while global architectures can waste resources on local patterns.Convolutions offer local fields, pooling hierarchies can expand them, and spectral methods and attention provide alternatives for global interactions.Source: §5.3 · pp. 62–63

5.4 Regular, unstructured and point-wise data

  • Grid-like connectivity supports convolutions even on deformed grids; graph message passing is equivalent to grid convolution, while point samples can use continuous kernels and neighborhood coarsening.Convolution stencils also connect grid-based neural networks to discretized differential operators.Source: §5.4 · pp. 63–64, §5.5 · p. 64

5.5 Hierarchies

  • Pooling hierarchies and dilation both expand receptive fields, but U-Nets can be much faster while requiring more implementation effort.U-Nets downsample and upsample, with skip connections restoring detail; dilation keeps the sample count constant and may have slower, less regular memory access. Graph networks favor multiscale hierarchies.Source: §5.5 · pp. 64–65

5.6 Spectral methods

  • Fourier Neural Operators offer global receptive fields, but their parameter scaling makes them unsuitable for 3D or higher-dimensional problems in the described comparison.The figure and text contrast FNO scaling with regular convolutions: exponents of 6 versus 5 in 3D; FNO frequency coverage also grows with domain size.Source: §5.6 · pp. 65–66, Figure 5.5 · p. 65

5.7 Attention and Transformers

  • Transformers use self-attention to model global dependencies, but standard attention scales quadratically with token count and dense token-space attention can discard spatial structure.Linear attention scales linearly in token count but produces a more approximate attention vector.Source: §5.7 · pp. 66–67

5.8 Summary of Architectures

  • The section recommends tailoring architecture to dependency patterns and considering data-structure transformations; it also notes hybrids and illustrates a U-Net surrogate for turbulent airfoil flows.The example replaces a full solver with a surrogate model from.Source: §5.8 · p. 67, §5.9 · p. 67

SUPERVISED TRAINING FOR RANS FLOWS AROUND AIRFOILS

These parts examine neural approaches to fluid-flow prediction and correction, from a supervised RANS surrogate for airfoils to a physics-constrained pressure network that makes flows divergence-free. They describe the models and training data, and evaluate performance on unseen airfoils and synthetic, out-of-distribution, and obstacle-flow cases.

6.1 Overview

  • The neural surrogate bypasses numerical RANS solvers to predict time-averaged velocity and pressure fields from airfoil shape, Reynolds number, and angle of attack.The section frames this as a widely used industrial application of Navier–Stokes solvers.Source: §6.1 · p. 69

6.2 Formulation

  • Inputs and outputs are both 3 × 128 × 128 grids: freestream velocity components and an airfoil mask map to pressure and two velocity components; field normalization is needed to prevent pressure scale from dominating the loss.The pressure field has approximately squared scaling with velocity, motivating normalization across the three target fields.Source: §6.2 · p. 69

6.3 Code coming up…

  • Training uses a small Spalart–Almaras RANS dataset generated in OpenFOAM, with 320 training and 80 validation samples sharing airfoil shapes but differing in conditions; the code is adapted from.New airfoil shapes are reserved for testing.Source: §6.3 · p. 70

6.4 RANS training data

  • The section recommends checking data dimensions, rough statistics, and content before training, and demonstrates visualizing a training sample’s inputs and outputs.The displayed channels are free-stream x/y velocity and mask as inputs, and pressure and x/y velocity as outputs.Source: §6.4 · pp. 70–71

6.5 Network setup

  • A fully convolutional U-net uses multiple spatial resolutions, a global receptive field, and encoder-to-decoder skip connections; its exponent-4 configuration has 585,027 trainable parameters.Validation predictions resemble targets but remain noisy near the airfoil boundary, attributed to the Dirichlet boundary; pressure is typically hardest to learn.Source: §6.5 · pp. 72, 74, §6.6 · p. 77
  • Validation loss fell from around 0.2 initially to ca. 0.02 after training, while the final logged training and validation losses were 0.01113 and 0.01996.The validation curve flattened after about 40 epochs without increasing, which the section identifies as avoiding a sign of overfitting.Source: §6.6 · p. 76

6.7 Test evaluation

  • Testing uses new, out-of-distribution airfoil shapes to assess generalization beyond validation data, and pressure is described as especially difficult because of cubic scaling with freestream velocity and localized peaks.The provided passages describe the test setup and challenge, but do not report a test-error result.Source: §6.7 · pp. 77–78
  • 0.026288176793605088 average relative test error shows the small, fast, differentiable network approximates the RANS solver, but misses localized pressure and y-velocity peaks.The L2 loss favors larger structures over localized peaks; the authors note the setup can improve with more data and changes.Source: §6.7 · pp. 79–80

7.1.1 Natural starting point

  • Begin supervised projects with a one-sample overfitting test, verify convergence, then increase data and model capacity while tuning the learning rate.A simple overfitting test helps reveal fundamental problems in code or data before more complex setups make them harder to find.Source: §7.1.1 · p. 81

7.1.3 Know your data

  • Track per-quantity statistics and histograms to find bad samples and outliers, and inspect relationships between quantities to understand dataset coverage.The airfoil visualization shows no extreme outliers, while the test samples cover the range of mean motions well.Source: §7.1.3 · p. 82, Figure 7.1 · p. 82

7.1.4 Where’s the magic? ￿

  • Neural networks approximate the training function rather than discovering general principles, so inputs at inference should resemble training inputs and solver interactions can cause distribution shift.Normalization and focusing on invariants can help shape the input space, but networks are not generally good at extrapolation.Source: §7.1.4 · p. 83

7.2 Supervised training in a nutshell

  • Supervised training is fast, stable, simple, and a good starting point, but it needs lots of data and can have suboptimal accuracy, generalization, and solver interactions.The supervised approach first approximates a solution manifold from examples, enabling rapid predictions once a sufficiently large dataset exists.Source: §7.2 · p. 83

8.1 Using physical models

  • Physical loss terms combine supervised targets with PDE residuals to anchor a solution, since either objective alone can fit samples poorly between points or fail to select a globally consistent solution.Residual constraints are soft: training may leave non-zero residuals, and there is no guarantee they reach zero.Source: §8.1 · pp. 87–88, PHYSICAL LOSS TERMS · p. 87

8.2 Variant 1: Residual derivatives for explicit representations

  • Explicit representations discretize residuals on a chosen mesh and differentiate through framework operations; a CNN can learn divergence-free projections without supervised samples and handle broad solution spaces.The example learns a pressure field whose gradient removes divergence from the input velocity field.Source: §8.2 · pp. 88–89

8.3 Variant 2: Derivatives from a neural network representation

  • Neural-field PINNs use fully connected networks and autodiff to compute spatial or temporal derivatives, making them suited to inverse problems but difficult for time dependence or broad solution manifolds.The approach is limited by ill-posed optimization and reconstruction cost, and is not well suited to the wide range of solutions in the airfoil example.Source: §8.3 · p. 89, §8.4 · p. 90

9.1 Solving Navier-Stokes

  • The pressure-projection step solves a Poisson problem, then subtracts the pressure gradient to produce a divergence-free velocity field.The preceding advection step is described as efficient and linear in the number of discretization points.Source: §9.1 · p. 91
  • The method predicts pressure from input divergence and trains through a discretized divergence constraint, without precomputed velocity-pressure pairs.Subtracting the predicted pressure gradient from the input flow yields the corrected velocity; an L2 loss penalizes its divergence.Source: §9.1 · p. 91, §9.3 · p. 96

9.2 Setting up the Discrete PDE

  • Training uses 100 random periodic divergent flow fields without known divergence-free counterparts; their input divergence averages around 0.4.The flows are generated with PhiFlow’s frequency-based synthesis.Source: §9.2 · pp. 92–93

9.3 Neural Network Training

  • A U-Net is chosen for the elliptic pressure problem because global communication matters; the described ResNet performs less well with its local receptive field.The network takes divergence as a single channel and outputs pressure; the described configuration has around 83k trainable parameters.Source: §9.3 · pp. 94, 96

9.3. Neural Network Training

  • The untrained network leaves a large divergence loss of 390.21857, while the training setup reports that loss should fall by more than two orders of magnitude after a few epochs.Training iterates over precomputed random fields and updates weights to reduce divergence loss.Source: §9.3 · p. 97, §9.4 · p. 98

9.5. Testing with New Inputs

  • The 2×-scale test inputs are out of the training distribution, and the learned pressure can differ in magnitude from the solver’s because constant offsets do not affect its gradient.The loss does not constrain absolute pressure values, so the network can produce valid pressure solutions with different offsets.Source: §9.5 · pp. 98, 100

9.5 Testing with New Inputs

  • In the reported comparison, Original mean divergence was 2059.143 and maximum divergence was 4.946; the network’s values were 11.510 and 0.417, while the solver’s were both 0.000.The network’s remaining divergence was larger than PhiFlow’s solver’s, but orders of magnitude smaller than the original.Source: §9.5 · pp. 99–100

9.6 Tougher Tests: Fluid Simulations with Obstacles

  • The tougher test introduces a buoyant smoke plume and an obstacle to check whether the network handles flow fields produced by a fluid solver.The supplied passage describes setting up the test but gives no outcome.Source: §9.6 · p. 100
  • The NN and classic solver produce remarkably similar final frames, while the NN reduces divergence in arbitrary flows despite training only on synthetic noisy inputs.The comparison ran for 50 simulation steps.Source: §9.6 · p. 102
  • The differentiable discrete operator supports rapid training convergence and zero-shot generalization to new inputs, including mostly correct pressure fields for unseen obstacles.The passage notes that adding obstacle geometry as an input and increasing loss values near boundaries could improve the implementation.Source: §9.6 · p. 102

NEXT STEPS

This section examines PINN-based reconstruction of Burgers’ equation using observed values, boundary conditions, and PDE residuals, along with limitations in the resulting solution. It outlines training, architectural, obstacle-handling, and solver-integration changes as possible directions for improvement.

NEXT STEPS

  • Future work includes improving network depth, features per layer, and other architectures, testing fully convolutional models on larger grids, and adding obstacle inputs with updated training data.Source: NEXT STEPS · p. 105

BURGERS OPTIMIZATION WITH A PINN

  • The reconstruction example uses Burgers’ equation with observations at t=0.5, zero Dirichlet boundaries, and a solution defined over t∈[0, 1].The solution is sampled at 128 points, while the neural network supplies its own discretization and permits sampling between those points.Source: BURGERS OPTIMIZATION WITH A PINN · p. 107

11.1 Formulation

  • The inverse problem fits a neural network to space-time solution samples while penalizing a PDE residual formed from network derivatives, following the PINN approach.The residual is intended to converge to zero.Source: §11.1 · p. 107

11.2 Preliminaries

  • The notebook represents the solution with an eight-hidden-layer, 20-unit tanh network and constructs the Burgers residual using time and space derivatives, including a second space derivative.The network is evaluated on a 128 × 33 space-time grid; its random initialization is smooth but is not yet a PDE solution.Source: §11.2 · pp. 108–110

11.3 Loss function and training

  • Training combines observed and boundary-value constraints with residual losses at interior points, but convergence is typically slow despite fast iterations.The example uses 100 points for each boundary constraint and 1,000 interior physics points.Source: §11.3 · p. 111
  • Training reportedly takes about 2 minutes and reduces error from roughly 0.2 to about 0.03, with the network converging toward a Burgers-equation solution.The notebook uses 10,000 iterations by default; increasing the iteration count may improve results.Source: §11.3 · pp. 111–112

11.3. Loss function and training

  • At the domain sides, the reconstruction fulfills the Dirichlet boundary conditions u = 0, but it does not represent the central shock at x = 0 well.Source: §11.4 · p. 113
  • The PINN’s reconstructed initial state is inaccurate, with extrema at x = ±1/2 far off and nonzero values at the x = ±1 boundaries.The text notes that improving the solution requires a surprisingly large number of iterations for this simple case.Source: §11.4 · pp. 113–114
  • 0.01136 was the mean absolute error for re-simulation across 16 steps; the text also characterizes the error as ca. 1.5⋅10−2 and significant for the simulation’s value range.Source: §11.4 · p. 117

11.5 Next steps

  • The quick PINN and soft-constraint setup is a starting point; next steps include tuning training, changing the architecture or optimizer, and extending the reconstruction interval by moving boundary conditions later in simulation time.The text cautions that more complex optimizers may not improve this simple example and says tighter solver-learning integration can substantially improve the inverse solve.Source: §11.5 · p. 118

TWELVE

This section explains differentiable physics as an approach that passes gradients through physical solvers, contrasting it with PINNs and illustrating it through advection and Burgers reconstruction problems. It also discusses analytic gradients for solver operations and the reported benefits and costs of differentiable physics for Burgers reconstruction.

DISCUSSION OF PHYSICAL LOSSES

  • Physical-loss methods enforce laws only as soft constraints; backpropagation computes derivatives but can make optimization costly, unreliable, and difficult to control.Each derivative requires backpropagation through the full network, especially costly for higher-order derivatives; the network may also need tricks to focus on the right solution regions.Source: DISCUSSION OF PHYSICAL LOSSES · p. 119, §12.2 · p. 120

12.1 Generalization?

  • PINNs typically encode one known solution, offer little out-of-range generalization, and require retraining from scratch for a different solution.Their samples come from the same known space-time domain, rather than representing test or out-of-distribution cases.Source: §12.1 · p. 119, §12.2 · p. 120

12.2 Summary

  • PINN representations are largely incompatible with classical numerical methods such as conjugate gradient, motivating differentiable solvers to bring those methods back into learning.The section also characterizes PINNs as yielding single solutions or narrow solution manifolds, unlike supervised neural surrogates and operators.Source: §12.2 · p. 120

INTRODUCTION TO DIFFERENTIABLE PHYSICS

  • Differentiable physics uses existing numerical solvers with gradients so learning can address larger classes of inverse problems, rather than only single inverse problems.The gradients let information flow between a simulator and a neural network, supporting improved learning feedback and generalization.Source: INTRODUCTION TO DIFFERENTIABLE PHYSICS · p. 123

13.1 Differentiable operators

  • A differentiable simulator is a sequence of operators that provide gradients with respect to their inputs, allowing reverse-mode learning through the solver.Because the loss is scalar, backpropagation can use Jacobian-transpose vector products rather than constructing and storing full Jacobians.Source: §13.1 · p. 124, §13.2 · p. 124

13.3 Learning via DP operators

  • Meaningful, monolithic solver operators avoid storing unnecessary intermediate states and allow tailored numerical methods, but require knowledge of the problem and may lack built-in gradients.The section notes that numerical techniques such as multigrid can be reused, while recommending that only task-relevant derivatives be provided.Source: §13.3 · p. 125

13.4.1 Introducing a specific advection scheme

  • For advection, the section formulates finding a velocity field as minimizing an L2 loss between the evolved and target densities, using first-order upwinding on a 1D Cartesian grid.The example is framed as a pure optimization task, not a neural-network model intended to generalize to unseen data.Source: §13.3 · p. 126, §13.4.1 · p. 127

13.4.2 Time evolution

  • For multiple advection steps, backpropagation chains Jacobian products to the initial state and accumulates velocity gradients across steps; the section also introduces Poisson’s equation as another example.The time-evolution explanation assumes a velocity field constant in time; Poisson’s equation is presented as a fundamental elliptic PDE arising in problems including electrostatics, gravity, and fluids.Source: §13.4.2 · p. 128, §13.5 · p. 128

13.5 Implicit gradient calculations

  • For pressure projection, the symmetric Poisson inverse lets the backward pass reuse the iterative solver without storing its internal states.The passage says this gives the same performance and accuracy as the forward solver; the broader lesson is to derive gradients analytically rather than blindly backpropagate through every forward operation.Source: §13.5 · p. 129

13.6 Summary of differentiable physics so far

  • Differentiable physics embeds discretized physical equations in deep learning, allowing neural networks to interact with solvers and use the physical model at inference.Unlike residual-based approaches, this can yield networks that robustly generalize to new inputs.Source: §13.6 · p. 130

GRADIENT

  • In the Burgers reconstruction example, the discretized solver determines system evolution, leaving the initial state as the unknown and eliminating the need for a neural network.The unknown initial state is represented by floating-point variables at the discretization’s sampling points.Source: GRADIENT · p. 131

14.1 Initialization

  • The simulation initializes a zero velocity field and a target constraint at t=0.5 (step 16), using periodic boundaries and spatial discretization Δx=1/128.The example uses a TensorFlow-backed PhiFlow setup.Source: §14.1 · p. 131

14.2 Gradients

  • The initial error is ca. 0.38 at simulation step 16; the L2 mismatch there supplies the loss used to compute gradients for the initial velocity.The gradient function evaluates the 32-step simulation and propagates the loss back to the initial state.Source: §14.2 · pp. 132–133

14.2. Gradients

  • The gradient provides a search direction for each velocity variable, which can be used to optimize the initial state to minimize the loss.Because gradients point toward increasing loss, optimization updates move in the opposite direction.Source: §14.2 · p. 134

14.3 Optimization

  • After five gradient-descent updates, the reconstructed state begins to show the target’s peaks, indicating that backpropagation through 16 solver steps drives the solution in the right direction.The example uses a learning rate of 5; the gradients can be chained through more complex sequences and longer look-ahead with minor code changes.Source: §14.3 · pp. 134–135

14.4 More optimization steps

  • Loss falls from 0.183476 at step 5 to 0.003263 at step 45 as gradient descent optimizes the initial state.The reported optimization ran for 132.33s.Source: §14.4 · p. 136
  • DP reduced error by about two orders of magnitude at comparable runtime because its gradients span all discretization points and time steps.Recovering the initial state is harder because the nonlinear model admits multiple solutions that satisfy the later-time constraints; the result is still close in this Burgers example.Source: §14.4 · pp. 136, 138

14.5 Physics-informed vs. differentiable physics reconstruction

  • MAE was 0.19298 for PINN versus 0.06382 for DP; DP was closer to ground truth overall but still missed some sharper features.The reconstruction visualizations also show that PINNs recover the overall shape but not the amplitudes as well.Source: §14.5 · pp. 139–141

15.1 Compatibility with existing numerical methods

  • PINNs compute derivatives through a neural-network representation, while DP ties learning to a numerical solver and discretization that can use established techniques with controllable accuracy and efficient evaluation.The PINN derivatives depend on the network representing the solution accurately enough to provide a useful optimization direction.Source: §15.1 · p. 143

15.2 Discretization

  • DP requires a suitable discretization that can impede or destabilize training if poorly chosen, while PINNs avoid an a-priori grid but construct an implicit discretization that is difficult to control.The Burgers example also shows PINNs struggling to propagate information backward in time.Source: §15.2 · pp. 143–144

15.3 Efficiency

  • Localized PINN sampling and weight updates can make boundary-condition fulfillment slow, whereas a well-chosen DP grid improves gradient flow and supports fast sampling from local neighborhoods.For fully connected PINNs, even one sample may require a full pass through the network.Source: §15.3 · p. 144

15.4 Efficiency continued

  • DP can cover larger solution manifolds, but longer look-aheads create more complex signals that require more computation per iteration and often take longer to converge.Each look-ahead may require many simulation steps and a similarly resource-intensive backpropagation pass.Source: §15.4 · p. 144

15.5 Summary

  • When the problem is known, a DP solver can improve neural-operator training and solution quality; the discussion identifies PINN limitations in performance and compatibility with numerical methods.The authors therefore focus on DP variants in subsequent chapters.Source: §15.5 · p. 145

SIXTEEN

This section explores combining differentiable physical solvers with neural networks to learn and correct simulated flows, including smoke-flow optimization, solver-in-the-loop wake-flow correction, and neural-operator approaches. It describes training through unrolled simulations and evaluating hybrid solvers on unseen Reynolds-number cases.

16.1 Physical Model

16.4 Batched simulations

  • Named inflow_loc batching runs four simulations with different smoke-inflow positions; the simulation with inflow at (16,5) serves as the reference for optimizing the other three initial velocities.Named dimensions broadcast matching tensor information across the batch and spatial dimensions.Source: §16.4 · pp. 148–149

16.5 Gradients

  • Gradients from the differentiable solver propagate the L2 smoke-density loss through 20 simulation steps to the initial velocity.The reference smoke field is treated as an external constant when computing the loss.Source: §16.5 · p. 150

16.6 Optimization

  • Loss fell from almost 300 to below 170, and optimized simulations aligned plume upper regions closely while retaining stem mismatches.The optimization used steepest gradient descent over 80 updates; fixed inflows and nonlinear evolution over 20 steps prevented exact matching.Source: §16.6 · p. 152, §16.7 · p. 155

16.8 Conclusions

  • The example extends differentiable physics to a 20-step Navier-Stokes solver, but remains direct inverse-problem optimization rather than a machine-learning task.It presents this setup as a basis for later combinations of differentiable physics and neural networks.Source: §16.8 · p. 155

INTEGRATING DP INTO NN TRAINING

  • In neural-network learning applications, a solver may be iterated with network evaluations in an autoregressive correction task, unlike pure prediction without a solver at inference.The section groups closure, homogenization or coarse-graining, and climate or weather parametrization under correction tasks.Source: INTEGRATING DP INTO NN TRAINING · p. 157, Figure 17.1 · p. 157

17.1 Switching the order

  • A PDE solver can generate randomized training inputs on the fly, improving input-distribution sampling and reducing data storage, but this ordering does not backpropagate through the solver.Because the solver is not influenced by the NN, it can instead be replaced by precomputed inputs or a loading function.Source: §17.1 · p. 158

17.2 Recurrent evaluation

  • In recurrent training, NNs modify states between solver steps and gradients pass backward through the full chain, exposing how NN actions affect later states and loss.Errors can grow exponentially across iterations, while recurrent Jacobians can cause vanishing or exploding gradients; the weight update sums contributions across time steps and propagates them through Jacobians.Source: §17.2 · pp. 158–159, §17.4 · p. 160

17.3 Composition of NN and solver

  • When an NN predicts a correction rather than a full state, a differentiable merge operator can reuse the current state; addition is a good starting point.The solver then advances the merged state, so the NN need only update parts that do not yet satisfy the learning objective.Source: §17.3 · p. 159

17.5 Backpropagation through solver steps

  • After 10 unrolled steps, training with solver gradients uses whole-course feedback to improve accuracy over supervised training, whose red curve deviates more from the reference than the differentiable-solver result.After 1024 simulation steps, both trained versions are stable and perform better than the unmodified solver; the figure compares Reynolds stresses and TKE.Source: §17.5 · pp. 160, 162, Figure 17.4 · p. 161

17.5. Backpropagation through solver steps

  • More than 14x longer simulation time was required to reach the turbulence-statistics accuracy achieved by the solver with a neural network in this case.The authors describe this as an initial data point for the practical performance gains possible after training.Source: §17.5 · p. 162

17.6 Alternatives: noise

  • Noise perturbations can reduce overfitting to training states, but because noise is undirected, solver-in-the-loop training is preferable when feasible to provide feedback from actual system evolution.The passage also points to denoising diffusion and flow matching as better-founded approaches for incorporating noise.Source: §17.6 · p. 162

17.7 Complex examples

  • The complex examples apply differentiable physics to learn numerical-simulation errors and to control a Navier–Stokes simulation over the long term using separate prediction and action networks.The error-learning example uses a KarmanFlow wake simulation around a spherical obstacle; these notebooks require more resources and may run longer.Source: §17.7 · p. 162, §18.3 · p. 165

18.1 Problem formulation

  • A neural operator can learn recurring discretization errors and apply local corrections to a coarse PDE solve, producing solutions closer to a higher-fidelity reference.The correction is trained against a reference discretization and must interact with the differentiable source solver, whose gradients account for evolving corrected states and avoid the distribution shift that can undermine supervised training.Source: REDUCING NUMERICAL ERRORS WITH NEURAL… · p. 163, §18.1 · pp. 163–164

18.3. Simulation setup

  • The marker field is passively advected for visualization, while the correction task focuses on velocity.The marker is not the fluid density.Source: §18.3 · p. 166

18.4 Network and transfer functions

  • A fully convolutional network takes x- and y-velocity plus Reynolds number as input and outputs a two-component velocity correction.The described network uses three input channels and two output channels.Source: §18.4 · p. 166
  • Transfer functions convert velocity between staggered PhiFlow grids and PyTorch tensors, while normalizing Reynolds number for the network and retaining its physical value for the solver.Training batches contain samples with the same Reynolds number.Source: §18.4 · pp. 167–168

18.5 Training setup

  • Training uses simulations 0–5 and reserves 6–9 for testing generalization to Reynolds numbers outside the training range.The setup uses Adam and a default batch size of 3; reference trajectories provide MSTEPS+1 time steps.Source: §18.5 · p. 169

18.6 Interleaving simulation and NN

  • Each training step unrolls simulation and network evaluations for MSTEPS steps, adds predicted velocity corrections, and minimizes sequence-wide L2 error; JIT compilation targets the simulation step.The simulation step includes an implicit pressure solve, making it a candidate for JIT compilation.Source: §18.6 · pp. 171–172

18.6. Interleaving simulation and NN

  • Each update unrolls four simulation and network steps: gradients skip the first solver, while the first network call in later steps receives feedback from three subsequent states.The feedback indicates whether its inferred correction moved those states in the right direction.Source: §18.6 · p. 173
  • Default training reduces loss from above 10 to around 1 and yields a source simulator paired with a network operator trained to interact with it for a chosen simulation domain.More epochs or larger and deeper networks can reduce the loss further.Source: §18.6 · p. 173

18.7 Test evaluation

  • Relative L2 errors were 0.10863638424314559 for the low-fidelity solver and 0.031211425131186844 for the corrected hybrid on an unseen Reynolds-number case.The evaluation describes the hybrid solver as generalizing to new Reynolds numbers.Source: §18.7 · pp. 175–176
  • During rollout, low-fidelity errors rise linearly, while hybrid errors grow more slowly and can remain low with larger networks and successful training.The hybrid relies on the base solver’s prediction, so a small correction can nudge it toward the target accuracy.Source: §18.7 · p. 176
  • The source simulation underestimates forming vortices and deviates more from the reference, while the learned solution stays closer with smaller, more evenly distributed errors.The passage describes this comparison in state and per-cell error images.Source: §18.7 · p. 178
  • Simple MAE or L2 norms may not reliably measure the hybrid solver’s improved behavior; fluid-specific metrics based on vorticity and turbulence could help.The text turns to another inverse problem with differentiable physics solvers in the next chapter rather than developing those metrics here.Source: §18.7 · p. 178

18.8 Next steps

  • Suggested training variants are msteps=1, which removes solver gradients and yields supervised training with substantially larger errors, or msteps=8 or 16, which may need a pretrained state to stabilize early iterations.The pretrained state effectively adds curriculum learning.Source: §18.8 · p. 178
  • Testing on tougher cases generated with external code can reveal that reduced training error does not always correspond to improved test performance.Source: §18.8 · p. 178

NINETEEN

This section explores differentiable-physics training of neural controllers for related inverse problems in incompressible-fluid simulations, progressing from generated data and supervised initialization to end-to-end control. It describes guiding a fluid marker to a target through a 16-step 2D Navier–Stokes simulation, with a reported relative MAE of about 5.45%.

SOLVING INVERSE PROBLEMS WITH NNS

  • The goal is to train a neural network to solve a larger class of related inverse problems, learning a solution manifold rather than optimizing each instance separately.The approach assumes sufficient similarity between problems, including continuity in the solution manifold.Source: SOLVING INVERSE PROBLEMS WITH NNS · p. 179
  • The test problem requires a space-time force control to drive an incompressible fluid so a passively advected marker reaches a target state, despite indirect constraints and many degrees of freedom.Because the control spans the full simulation, the controller must anticipate how state changes affect later behavior.Source: SOLVING INVERSE PROBLEMS WITH NNS · p. 179

19.1 Formulation

  • The formulation pairs a predictor OP that plans a desired state with a CFE actor that applies forces, using differentiable-physics gradients to train the nonlinear coupled system.The model uses discretized two-dimensional incompressible Navier–Stokes dynamics and transports a marker density.Source: §19.1 · p. 180

19.3 Data generation

  • Training data pairs randomly placed starting and target shapes; because these examples lack intermediate frames, a separate moving-rectangle dataset supplies sequences for supervised initialization.The target dataset has 1,000 examples, while the separate dataset provides intermediate frames for pretraining.Source: §19.3 · pp. 181–182

19.4 Supervised initialization

  • Supervised initialization trains OP networks for horizons of 2, 4, 8, and 16 steps to provide rough motion planning before end-to-end refinement.The 1,000 samples are split into 800 training, 100 validation, and 100 test samples.Source: §19.4 · p. 183

19.5 CFE pretraining with differentiable physics

  • CFE pretraining uses a single differentiable-solver step, while the network learns forces between pairs of states without simulating a training sequence.Source: §19.5 · p. 184

19.6 End-to-end training with differentiable physics

  • Joint end-to-end training of the CFE and OP networks through the differentiable fluid solver is presented as the key step and as yielding significantly improved control quality over supervised pretraining.The networks are initialized from supervised checkpoints before being trained together.Source: §19.6 · pp. 184–185

19.6. End-to-end training with differentiable physics

  • The trained network guides marker density to the target location and shape through a full 2D Navier–Stokes simulation over 16 time-integration steps, keeping motion coherent despite potentially chaotic forces.The network is described as pushing and deforming the fluid marker to match the target.Source: §19.6 · pp. 186–187
  • 0.05450168251991272 relative MAE was achieved, corresponding to the guide’s estimate that more than about 94% of marker density ends up in the right spot.The error compares the final density configuration with the initial density; the text describes the standard training setup as yielding a 5–6% relative residual error.Source: §19.6 · p. 187

19.7 Next steps

  • Testing with new target shapes can assess controller generalization, while using a RefinedSequence instead of a StaggeredSequence is suggested to improve control and reduce density error.The refinement approach is described as prediction refinement.Source: §19.7 · p. 187

TWENTY

The section connects differentiable physics and learned models with probabilistic learning for uncertainty-aware inference and distribution modeling. It covers normalizing flows, including RealNVP and continuous ODE-based approaches, their transformations and solver choices, and the computational cost of likelihood evaluation at scale.

20.1 Integration

  • Differentiable-physics training couples classical numerical solvers with learned components, enabling hard constraint enforcement and improved forward or backward solves.The coupled solver can project and enforce constraints as classical solvers do.Source: §20.1 · p. 189, §20.3 · p. 190

20.2 Reducing data shift via interaction

  • Training-time PDE interaction generates trajectories on the fly and provides solver feedback, reducing data shift and improving resilience to unseen inputs.The network receives directed feedback during solver iterations rather than relying only on an a-priori training distribution.Source: §20.2 · p. 189

20.3 Generalization

  • Hybrid solvers handle large-scale distribution shifts while learned models focus on localized structures; DP inherits selected methods’ efficiency and accuracy but requires gradient-capable simulators and framework support.The PDE solver handles broad input-distribution changes that learned models may not generalize to on their own.Source: §20.3 · p. 190

21.1 Uncertainty

  • Predictive uncertainty motivates probabilistic models because measurements, models, discretizations, and learned predictors all introduce error; aleatoric data noise and epistemic model uncertainty can be difficult to disentangle.The probabilistic setting represents multiple possible solutions and their probabilities rather than one deterministic output.Source: §21.1 · pp. 193–194, INTRODUCTION TO PROBABILISTIC LEARNING · p. 193

21.3 Simulation-based Inference

  • For inverse problems, simulation-based inference targets posterior distributions, and deep learning can train a conditional density estimator from simulations alone despite the difficulty of computing likelihoods.Marginal likelihoods can be intractable when latent variables are difficult to sample; learned SBI methods require costly upfront training and lack rigorous theoretical guarantees.Source: §21.3 · pp. 194–195

22.1 Fundamentals: A Training Objective

  • Minimizing KL divergence to a target distribution reduces to minimizing the learned density’s negative log-likelihood on samples from that target.The target-density term is independent of the model parameters, so training can use samples from the target distribution.Source: §22.1 · p. 197

22.2 From Unconditional to Conditional

  • A conditional posterior can be learned by sampling inputs from the prior and observations from the simulator, provided the proposal is a valid density.For Gaussian densities, negative log-likelihood training is equivalent to minimizing L2 error.Source: §22.2 · p. 198

22.3 Learning Distributions with Normalizing Flows

  • Normalizing flows transform a simple base distribution into a complex target through invertible, differentiable mappings, with Jacobian determinants determining the transformed density.The change-of-variables relation also makes log-likelihood training convenient.Source: §22.3 · p. 198
  • Normalizing flows sample by mapping draws from a base distribution through a sequence of transformations and compute target densities using the inverse mappings and their Jacobians.Source: §22.3 · p. 199

22.4 Practical Example: Learning Gaussians

  • A 2D Gaussian-mixture example provides a target distribution for sampling and likelihood evaluation, with components that can have different means and variances.Source: §22.4 · pp. 199–201

22.5 A Simple Normalizing Flow based on Affine Couplings

  • The RealNVP demonstration uses six affine-coupling blocks built from three-layer fully connected networks with ReLU activations to learn the toy mixture.Source: §22.5 · p. 201

22.5.1 Setup Dataset and Train the Normalizing Flow

  • Training uses samples from the Gaussian mixture; the example generates 50,000 samples and trains the model for 50 epochs with batches of 128.Source: §22.5.1 · pp. 203–204

22.5.2 Visualizing the Likelihood of the Trained Normalizing Flow

  • The trained RealNVP’s likelihoods qualitatively approximate the Gaussian-mixture reference, with training loss ending at 1.9213 after 50 epochs.Source: §22.5.1 · p. 205, §22.5.2 · p. 206

22.5.3 Visualizing Different Layers

  • Intermediate flow layers take arbitrary, initialization-dependent shapes because they receive no gradients about the final distribution and have no constraints, provided the final state matches the target.Source: §22.5.3 · p. 208

22.6 Neural ODEs: Making Normalizing Flows Continuous

  • Neural ODE flows are proposed to address the invertibility and fixed-layer-count limitations by continuously transforming distributions over artificial time from 0 to 1 using a learned velocity predictor.Source: §22.6 · pp. 208–209
  • A continuous normalizing flow replaces a sequence of discrete steps with an ODE that queries velocity from t=0, a normal Gaussian distribution, to t=1, the target distribution.The ODE solve also has an analytic formulation for backpropagation gradients.Source: §22.6 · p. 209

22.6.1 Building a Continuous Normalizing Flow

  • The FFJORD architecture is used to implement the continuous normalizing flow.Source: §22.6.1 · p. 209

22.6.2 The Continuous Normalizing Flow Network

  • The network integrates its neural velocity estimator with a differentiable ODE solver, allowing loss gradients to backpropagate to FFJORD weights.The implementation uses odeint from torchdiffeq.Source: §22.6.2 · p. 210

22.6.3 Training

  • Training replaces discrete step-by-step transformations with continuous integration, and loss decreased from 4.8563 at Epoch 1/50 to 1.9453 at Epoch 50/50.Source: §22.6.3 · pp. 212–213

22.6.4 Visualization of the Continuous Normalizing Flow

  • The continuous-time flow transforms the Gaussian sampling distribution to the target much more smoothly than the discrete-layer implementation.The figure compares the transformations over time.Source: §22.6.4 · p. 215

22.7 Summary of Normalizing Flows

  • Neural ODEs allow different velocity-network architectures and solver step sizes, trading speed against accuracy, but likelihood evaluation requires many network evaluations and is computationally expensive.The ODE must be solved from t=1 to t=0 to evaluate likelihoods, making scaling to high-dimensional data and large networks difficult.Source: §22.7 · p. 215

TWENTYTHREE

This section presents score-based generative modeling, learning local log-density gradients without normalized likelihoods and using them for Langevin sampling, illustrated with a Gaussian-mixture toy dataset whose reference scores are directly computable. It then describes Annealed Langevin Dynamics, which moves from larger to smaller noise scales to address the tradeoff between guidance and sample quality.

23.1.1 Visualizing Samples, Likelihoods and Scores

  • A Gaussian-mixture toy dataset provides directly computable reference scores, and a helper visualization displays samples, score fields, and likelihood contours.Source: §23.1 · p. 217, §23.1.1 · p. 218

23.1. Gaussian Toy Dataset with Analytic Scores

  • Because scores depend on local likelihood gradients rather than globally normalized densities, they are easier to learn than flexible normalized models, while normalizing flows can be computationally expensive and scale poorly to high dimensions.The score is the gradient of the log likelihood, ∇_x log p(x).Source: SCORE MATCHING · p. 217, §23.1 · p. 220

23.2.3 Compare Learned and Reference Scores

  • The learned score is evaluated qualitatively by plotting its predicted vector field beside the Gaussian mixture’s ground-truth score field.Source: §23.2.3 · p. 223

23.2 Learning the Score

  • Gaussian-noise perturbations make conditional score targets computable as -noise/sigma^2, enabling practical score learning when sigma is chosen suitably.The perturbation is x̃ = x + σz with z drawn from a standard Gaussian; the noise should permit gradient computation without excessively distorting the target distribution. The toy implementation uses a four-hidden-layer score network.Source: §23.2 · pp. 220, 222, 224, §23.2.1 · p. 221

23.3 Langevin Dynamics

  • Langevin dynamics updates samples along the score while adding Gaussian noise; under regularity conditions, iterates converge to target-distribution samples as K→∞ and ε→0.The illustrated implementation uses x ← x + step_size·score + sqrt(2·step_size)·noise.Source: §23.3 · p. 224, §23.3.1 · p. 225

23.3.2 Visualize the Trajectories

  • Trajectory visualization starts points on a dense regular grid and colors them by their initial positions to show where they move across the target density landscape.Source: §23.3.1 · p. 224, §23.3.2 · p. 225

23.3. Langevin Dynamics

  • Regularly spaced samples move toward higher-density regions, with reddish samples mainly reaching the right cluster and the other colors reaching the central peak.The trajectory visualization overlays samples on the mixture density contours.Source: §23.3 · p. 227

23.4 Annealed Langevin Dynamics

  • Using multiple noise scales from largest to smallest resolves the single-noise tradeoff by providing gradients broadly at high noise, then refining samples at lower noise.The score network receives the noise scale as an additional input.Source: §23.4 · p. 227

23.4.1 Network and Dataset Definition with Noise Level

  • The noise-aware network takes σ alongside each input, while training examples add Gaussian noise and use the target score -noise/σ^2.Sampling decreases σ step by step and uses the noise-aware score network for gradients.Source: §23.4.1 · p. 228, §23.4.2 · pp. 229–230

23.4.3 Visualization of Trajectories

  • Annealing moves samples from wide, smoothly varying score contours toward sharper target-distribution peaks and removes the need for a manually prescribed perturbation.The visualization shows this progression across noise scales.Source: §23.4.3 · p. 231

23.5 Score Summary

  • Score matching scales to high-dimensional data without backpropagating through many steps or requiring non-trivial prior distributions.The summary contrasts its scalability with CNFs or NeuralODEs.Source: §23.5 · p. 232
  • The method still requires a carefully chosen noise-scale sequence, lacks convenient maximum-likelihood training, and can make inference expensive through many score evaluations.The text says how to obtain a good noise-scale sequence remains unclear.Source: §23.5 · p. 232

TWENTYFOUR

This section examines denoising diffusion and flow matching as generative approaches for learning conditional solution distributions, with examples involving toy posteriors and a RANS airfoil application. It outlines their accuracy and inference-cost tradeoffs, including reported flow-matching performance with fewer steps than DDPM.

24.1 Latent Variable Models

  • The marginal likelihood is intractable, so training uses a variational lower bound whose tractable Gaussian KL term motivates predicting added noise instead of the mean.Removing the first β^2 factor gives a simplified denoising loss that increases weighting for samples with small β near the end of the reverse chain.Source: §24.1 · pp. 233–234

24.2 Full Denoising Algorithm

  • DDPM’s simple, stable L2 denoising loss supports supervised training and probabilistic inference, but generating samples requires many sequential network evaluations.The method can learn distributions with multiple possible outcomes, while inference can be orders of magnitude more expensive than deterministic networks.Source: §24.2 · pp. 235–236
  • Training samples a timestep and noisy data point for noise-prediction learning, whereas inference starts from pure noise and sequentially denoises across T steps.Original DDPM work used T = 1000; reducing T by about an order of magnitude still leaves around 100 network evaluations.Source: §24.2 · pp. 235–236, §24.3.4 · p. 241

24.3.1 Define the Forward Process

  • In the Gaussian-mixture demonstration, forward noise makes the two peaks begin disappearing after about 200 steps and visually merge into one peak centered at zero.Source: §24.3.1 · pp. 237, 239

24.3.2 Dataset and Dataloader

  • The training example precomputes noisy samples from the mixture and uses a three-hidden-layer network conditioned on time to predict the added noise.Training uses an L2 loss; the logged loss was 0.1529 at epoch 1 and 0.0624 at epoch 100.Source: §24.3.2 · pp. 239–240, §24.3.3 · pp. 240–241

24.3.4 Inference with the Reverse Process

  • The Gaussian-mixture inference example starts from pure noise and performs the full 1,000-step reverse process to produce a target-distribution sample.Source: §24.3.4 · pp. 241–242

24.3.5 Visualize the Sampled Trajectories

  • The DDPM network captures both distribution peaks with simple, stable training, but producing samples requires many costly sequential network evaluations.Even 10 neural function evaluations make it 10x slower than the regular deterministic version.Source: §24.3.5 · p. 244
  • Further DDPM extensions include faster inference, conditioning on external parameters and observations, and incorporating PDE-based constraints for physics simulations.The DDPM discussion identifies computational cost from repeated sequential evaluations as the remaining caveat.Source: §24.3.5 · p. 244

25.1 Learning Flows with Velocities

  • Flow matching trains a neural velocity field to map samples from p0 to p1, using tractable regression over easy-to-sample conditional paths specified by a latent variable.The flow is represented as an ODE, and the conditional flow matching objective can be used for actual training.Source: §25.1 · pp. 245–246

25.2 Mappings and Conditioning

  • Choosing linear paths toward target samples gives flow matching straight trajectories that can often be integrated in far fewer steps than denoising diffusion.Paths may not be perfectly straight, so one step can be suboptimal and further fine-tuning or distillation may be needed; the method aims to preserve accurate posterior coverage.Source: §25.2 · p. 246, Figure 25.1 · p. 247
  • The endpoint-conditioned mapping transports standard Gaussian samples along linear trajectories while reducing their standard deviation from 1 to σmin, preserving a continuous distribution at the endpoint.For this coupling, the transport path coincides with optimal transport between two Gaussian distributions.Source: §25.2 · p. 246
  • Unlike normalizing flows, this approach need not preserve densities and has a tractable learning objective that can be trained in single steps rather than backpropagating through a full Neural-ODE chain.The velocity viewpoint also allows flexibility in network architecture, while simple prescribed velocities support faster training and inference.Source: §25.1 · p. 245, §25.2 · p. 246

25.3 Implementing Flow Matching

  • The implementation trains on sampled times and supervised straight velocities without a noising schedule, then solves the learned ODE using 100 default steps—an order of magnitude fewer than the DDPM example.The passage notes that fewer than 100 steps may suffice in practice.Source: §25.3 · pp. 247–248, §25.3.1 · p. 249, §25.3.2 · p. 250

25.3.3 Visualize the Sampling Trajectories

  • Flow matching represents both density peaks in the toy posterior and yields accurate posterior samples with relatively mild computational overhead compared with a deterministic model.The passage describes the visualized sampling trajectories as supporting this result.Source: §25.3.3 · p. 252

DENOISING AND FLOW MATCHING SIDE-BY-SIDE

  • The RANS airfoil task uses solutions that may oscillate at larger Reynolds numbers, and applies denoising and flow matching to a forward problem from initial conditions x to solution y.Arbitrary ground-truth data are used so the learned target distribution can be quantified; the setup differs from simulation-based inference, which seeks x for an observation y.Source: DENOISING AND FLOW MATCHING SIDE-BY-SIDE · p. 253

26.1 Intro

  • Denoising models learn to restore signals from noise, whereas flow-based approaches transform distributions; the text says flow matching trains more stably and enables inference orders of magnitude faster through straighter paths.The text also says an existing denoising-model setup can be converted to flow matching surprisingly simply.Source: §26.1 · p. 253

26.2 Problem statement

  • The generative model should learn the conditional distribution p(y|x), marginalizing over hidden state ψ, so repeated evaluations produce efficiently sampled solutions with the appropriate probabilities.The example hidden state includes a non-converging RANS solver solution.Source: §26.2 · p. 254

26.4 Denoising with Diffusion Models

  • DDPM gradually noises a solution toward a Gaussian distribution, then learns to predict that noise from the noisy solution, condition x, and time t using an MSE loss.The implementation uses a cosine beta schedule; its training log reports 10,000 epochs and 50,000 total iterations.Source: §26.4 · pp. 255–256, §26.5 · pp. 256, 258

26.6 Flow Matching

  • Flow matching learns the time derivative of a differentiable transformation between distributions as a time-dependent vector field, training a neural network with an L2 loss between predicted and target velocities.The path starts from a simple distribution such as Gaussian noise and ends at the target data distribution.Source: §26.6 · p. 259
  • Conditional flow matching uses known x1 samples to marginalize conditional densities and vector fields, making conditional-flow learning mathematically equivalent to learning the original flow.Source: §26.6 · p. 259
  • The chosen optimal-transport path reduces Gaussian spread toward sigma_min while moving its mean from zero to x1, yielding a time-independent vector field and potential single-Euler-step inference.Source: §26.6 · p. 259

26.7 Implementing Flow Matching

  • The implementation interpolates x0 and x1 with sig_min, targets constant velocity x1-(1-sig_min)*x0 in cfm_loss, and reuses the denoising U-net for a fair comparison.Source: §26.7 · pp. 260–261

26.8 Test Evaluation

  • For one test input, the flow-matching comparison varied integration counts from 1 to 100 and used 100 noise-start samples per count to estimate mean and standard-deviation fields.Source: §26.8 · p. 263
  • 20-step flow matching did well on standard-deviation fields at comparable accuracy to 200-step DDPM, making it effectively 10x faster; 1-step flow matching failed on standard deviation.On the same test case, the mean was relatively easy to reproduce for all methods, and even 1-step flow matching mostly got it right, though with some noise.Source: §26.8 · p. 265

26.9 Quantified Results

  • The quantified evaluation covers six Reynolds numbers, with the middle four interpolating training parameters and the first and last extrapolating, and plots predicted against ground-truth standard deviations.Source: §26.9 · pp. 265–266

26.9. Quantified Results

  • 5- and 20-step flow matching already perform very well and are largely on par with the DDPM variant, while 1-step flow matching performs poorly.The estimates also capture extrapolation regions reasonably well, and the high-Re case is handled very well.Source: §26.9 · p. 267
  • The networks overestimate variance in low-Re cases because training data lacked truly static cases.Despite this, they capture the extrapolation regions reasonably well.Source: §26.9 · p. 267
  • Flow matching yields excellent posterior estimates and samples with very few integration steps, making it a contender for probabilistic simulations.The setup spans posterior distributions from almost constant to strongly varying across Reynolds numbers; practical surrogates must also be fast and use reasonable hardware resources.Source: §26.9 · p. 267

26.10 Next steps

  • Test whether higher-order integration methods improve on Euler, counting every neural-network call within each integrator.The passage gives RK4's four neural-network calls as an example.Source: §26.10 · p. 267
  • Improve trained-model accuracy by increasing the number of epochs and the U-net feature count.Source: §26.10 · p. 267
  • Try additional rectification steps from Liu et al. to potentially reduce the number of required steps further.Source: §26.10 · p. 267

TWENTYSEVEN

This section examines physics feedback for improving learned inverse-problem samples, covering inference-time guidance for flow matching and physics-integrated score matching, their conditions and limitations, and examples in gravitational lensing and heat diffusion. It also describes a reconstruction–spectral-error tradeoff in the SMDP heat example and recommends diffusion modeling when solutions are distributions not well represented by their mean.

27.1 Guiding Diffusion Models

  • Physics priors are more promising at inference than training: learned models are limited by model and data size, while simulator gradients can guide samples beyond the model’s accuracy plateau.This addresses the lack of a direct feedback loop between generative networks, observations, and samples that makes precise physics constraints difficult to impose through learning alone.Source: §27.1 · pp. 269–270, INCORPORATING PHYSICAL CONSTRAINTS · p. 269

Theoretical correctness

  • Naive simulator-gradient steps can distort posterior samples, but using the estimated final state only as control input and retaining vanilla flow-matching loss preserves theoretical properties unlike likelihood guidance.The distortion could pull distinct posterior samples toward a single simulator-favored solution.Source: §27.1 · p. 270, Theoretical correctness · p. 273

27.2 Physics-Guided Flow Matching

  • A small control network combines a pretrained flow with simulator feedback; freezing the pretrained network typically preserves performance while reducing added training compute.The chapter reports that simulator feedback gains cannot be achieved merely by training on larger datasets.Source: §27.2 · p. 270

Time-dependence

  • Because early flow states are noisy and poor simulator inputs, the method extrapolates them to an estimated final state and applies control only for t ≥ 0.8.The estimated state becomes more reliable later in the flow process; below the threshold, the pretrained flow is used directly.Source: §27.2 · p. 271, Time-dependence · p. 273

27.2.1 Physics-based Controls

  • Controls use a differentiable simulator’s cost and gradient or a learned encoder when the simulator is non-differentiable; stochastic-simulator noise is sampled per training step and fixed per inference trajectory.The differentiable-simulator cost can measure the mismatch between the estimated solution and observation, for example with an L2 loss or likelihood.Source: §27.2.1 · p. 272, §27.2.2 · p. 273

27.2.3 An Example from Astrophysics

  • FM with a simulator yields an average χ² of 1.48 versus 1.74 for the best classic baseline, AIES, and is about 35× faster than AIES.The standard FM variant is on par with classic inverse solvers, while the simpler DPS combination does not match their accuracy.Source: §27.2.3 · p. 273, Figure 27.3 · p. 274

27.3 Score Matching with Differentiable Physics

  • SMDP combines an approximate inverse physics simulator with a learned correction to reverse the system; single-step correction training is equivalent to score matching, while longer recursive training relates to maximum-likelihood training of a probability flow.The resulting solver offers posterior sampling and is described as accurate and temporally stable.Source: §27.3 · p. 274, §27.3.1 · p. 275

27.3.2 SMDP in Action

  • SMDP’s ODE trajectories have the lowest reconstruction MSE, while SDE solutions significantly improve spectral error and the version without the reverse physics step has larger spectral error.The figure reports metrics averaged over three runs.Source: Figure 27.7 · p. 276
  • ODE inference achieves the best reconstruction MSE, whereas SDE inference performs very well in spectral error and yields visually convincing solutions with only a slight MSE increase.The ODE solutions are smooth and lack necessary small-scale structures.Source: §27.3.2 · p. 277
  • In SMDP, inference noise supplies entropy needed to synthesize small-scale structures, and ODE and SDE tradeoffs occur with identical weights.The repository also shows an inverse Navier–Stokes solve.Source: §27.3.2 · p. 277

27.4 Summary of Physics-based Diffusion Models

  • PDE-based constraints in diffusion modeling rely centrally on physics-solver gradients and concepts such as unrolling, while probabilistic modeling adds complexity but provides distributions of solutions.The methods also incur costs on training and inference.Source: §27.4 · p. 277
  • Use diffusion when solutions form distributions not well represented by their mean; if the mean is acceptable, regular neural networks reduce training and inference complexity.For varied solutions, flow matching is robust, builds on deterministic training, and adds only mild computational overhead.Source: §27.4 · p. 277

TWENTYEIGHT

This section explores physics-based diffusion methods for simulation prediction, from SMDP score matching and reverse-time sampling to conditional autoregressive diffusion for turbulent-flow time prediction. It covers model conditioning, training and sampling workflows, evaluation across physical conditions, and qualitative, spectral, probabilistic, stability, and computational considerations.

SIMULATIONS

  • The notebook integrates a differentiable simulator with SMDP to combine physics and score matching for inverse problems.It also illustrates score-function training via log likelihoods in a reduced problem.Source: SIMULATIONS · p. 279

28.2 Implementation Overview

  • The implementation generates data, trains with a sliding window, visualizes the learned score, and performs inference with the probability flow ODE and reverse-time SDE.The core training and inference algorithms can be modified for the variants discussed in the paper.Source: §28.2 · p. 279

28.4. Visualization of SDE paths

  • The toy SDE trajectories start at 1 or −1 and approach zero stochastically.The initial values are drawn with equal probability from a categorical distribution; the system uses λ = 7 and g = 0.03.Source: §28.1 · p. 279, §28.3 · p. 280, §28.4 · p. 282

28.5 Generate Training Data

28.5.1 Neural Network Setup

  • The score is approximated by an MLP with ELU activations and five hidden layers of 30, 30, 25, 20, and 10 neurons.The probability-flow-ODE solver uses Euler steps and computes an L2 loss against the training trajectories.Source: §28.5.1 · pp. 284–285

28.5.2 Visualizing the Score Field

  • Before training, the score function shows random but smooth transitions.The notebook samples the score on a regular grid for visualization.Source: §28.5.2 · p. 285, §28.5 · p. 287

28.6 Training and Sliding Window Method

  • The sliding-window method starts with ROLLOUT_start and increases the window by ROLLOUT_add every steps epochs for ROLLOUT_increases times.It subsamples points on each trajectory to reduce their number.Source: §28.6 · p. 287

28.6.1 Single step loss

  • The single-step score training produces recognizable trajectory bands, while lower-learning-rate training on uns subsampled data typically yields a cleaner score field.The first stage uses 2500 epochs and a sliding window of size 2; the next stage lowers the learning rate and uses all training samples.Source: §28.6.1 · pp. 288–291

28.7.1 Reverse-time SDE

  • Reverse-time SDE samples show noisy initial motion and reliably converge to 1 and −1 with default settings.Reducing accuracy or training duration can produce erroneous score regions that send particles off track before t=0.Source: §28.7 · p. 294

28.7.2 Probability flow ODE

  • The probability flow ODE removes the SDE’s Brownian diffusion term, producing mostly horizontal motion that bifurcates toward the two endpoints.Source: §28.7.2 · p. 294, §28.7 · p. 296

DIFFUSION-BASED TIME PREDICTION

  • Conditional diffusion models provide distribution-based autoregressive rollouts, motivated by uncertainty and unobserved state influences that deterministic simulators do not capture.The notebook proposes evaluating accuracy on transonic fluid flow using statistics from a simulated reference.Source: DIFFUSION-BASED TIME PREDICTION · p. 297
  • Diffusion-based time predictors are reported to remain unconditionally stable under autoregressive rollout; the training approach uses DDPM, though flow matching could also work.The passage attributes stability to learning to handle perturbations and accumulated state errors.Source: DIFFUSION-BASED TIME PREDICTION · p. 298

29.1 Conditioning

  • For conditioning, the model denoises conditioning inputs alongside targets during training, then recomputes known conditioning values at each reverse step to prevent drift.Treating conditioning like diffusion outputs encourages the model to incorporate conditioning features and can reduce error accumulation.Source: §29.1 · p. 298
  • Training jointly denoises consecutive states, while inference fixes the previous state x_t as conditioning and predicts x_{t+1}; physical time is separate from denoising time.A variable Δt can also be supplied as a conditioning parameter, though the guide fixes it to 1 for simplicity.Source: §29.1 · pp. 298–299, Figure 29.2 · p. 299

29.2 Implementation

  • The ACDM implementation targets probabilistic, accurate next-step prediction for turbulent-flow simulations and uses a pretrained checkpoint to shorten training.The notebook also describes installing einops and the PBDL-Dataloader, with separate instructions for local runs.Source: §29.2 · p. 299

29.3 Backbone Network Definition

  • The modern U-Net replaces conventional skip connections with attention mechanisms, uses GELU and group normalization, and combines ConvNeXt-based residual blocks with diffusion-time embeddings.Softmax attention is confined to the bottleneck because it scales quadratically with input size; linear attention is used for skip paths.Source: §29.3 · pp. 300–301

29.4 Variance Schedule

  • The DDPM noise process uses an adjusted linear beta schedule intended to work with more than 10 diffusion steps.The implementation warns that schedules with fewer than 10 steps require adjustments.Source: §29.4 · p. 305

29.5 Diffusion Model Definition

  • The diffusion model registers DDPM process hyperparameters as buffers so they remain fixed rather than being learned.Its setup computes the forward-noising and posterior quantities used by the diffusion process.Source: §29.5 · p. 305, §29.4 · p. 306

29.4. Variance Schedule

  • The schedule computes posterior variance for q(x_{t-1}|x_t,x_0) from beta values and cumulative alpha terms.Source: §29.4 · p. 306
  • The forward step adds noise and predicts it during training; inference iterates reverse diffusion, jointly denoises conditioning and data, then restores ground-truth conditioning to focus sampling on data.The inference loop adds random noise only for non-final predictions.Source: §29.4 · pp. 306–307

29.5. Diffusion Model Definition

  • Training uses simulations 20 and 21 and testing uses simulation 22, while manually setting normalization constants from the full dataset avoids downloading all 5GB.The dataloader uses normalization constants from the full dataset despite loading only three simulations.Source: §29.4 · p. 307

29.6 Training

  • 0.0011557 was the reported [Epoch 9, FULL] loss at training completion.Training can continue from the provided checkpoint in 10 epochs taking less than a minute, or start from scratch, which takes hours and yields noticeably worse prediction quality and diversity because the example dataset is limited.Source: §29.6 · pp. 309, 312

29.7 Test Dataset

  • The test set uses simulation 22 at a different physical time and a larger Mach number than the training data, which used Ma = 0.72.The test dataset is intended to check performance on sequences outside the training data.Source: §29.7 · p. 312

29.8 Test Inference

  • Test inference samples probabilistic predictions for flow trajectories and stores predictions and ground truth in tensors indexed by samples, sequences, sequence length, channels, and spatial dimensions.The stated tensor shape is samples × sequences × sequenceLength × channels × sizeX × sizeY.Source: §29.8 · p. 313
  • Test inference forms predictions from prior input steps and uses a temporal stride of two for compatibility with the time step.The inference code adds the Mach number to target data before combining it with the input.Source: §29.8 · pp. 313–314

29.9 Accuracy of the Prediction

  • ACDM closely matches early flow states but produces realistic variation by t=60, where unstable cylinder shock waves permit differing predictions.Different inference noise values and longer rollouts produce further variations.Source: §29.9 · p. 315

29.9.1 Temporal Stability

  • Temporal-derivative comparisons are expected to become smoother with more sequences and samples, while additional training data can improve results and posterior samples should vary more than individual simulations.The section compares temporal derivatives of ground-truth trajectories and ACDM predictions.Source: §29.9.1 · p. 316

29.9.2 Spectral Analysis

  • The trained ACDM matches the ground-truth spectrum at low and high frequencies, smooths several downward spikes, and captures its statistics well.The spectrum is evaluated at a point downstream of the cylinder.Source: §29.9.2 · p. 317, §29.9 · p. 318

29.10 Summarizing Time Predictions with Diffusion Models

  • ACDM provides posterior samples of PDE time evolution, reproducing dataset changes and uncertainties as multiple possible solutions.For non-trivial data, it can preserve ambiguities as a distribution and retain reference-data statistics over long rollouts; stable models in the tests did not diverge over several hundred thousand rollout steps.Source: §29.10 · p. 319
  • Diffusion models are more computationally expensive than classic methods and generally do not improve mean-solution prediction, so they offer little payoff for unique input-output mappings.The models are run repeatedly to produce a sample, though flow matching could reduce the inference cost by a small factor.Source: §29.10 · p. 319

UNCONDITIONAL STABILITY

This section examines stable long-term prediction and practical training choices, then introduces graph-based diffusion models, including latent-space variants, for learning and sampling physical-state distributions from limited trajectories. It evaluates their generated distributions on flow and ellipse tasks, including pressure distributions, and considers sampling efficiency and the limits of single-sample accuracy.

30.2 Comparing Architectures

  • On Inc-high, all methods and runs stayed stable for 200,000 steps, while on shock-rich Tra-ext one U-Net-tn failed early and U-Net-ut and ACDM remained stable across runs.The 200,000-step horizon is not proof of infinite stability; the tests extrapolated to Reynolds numbers 960–1000 for Inc and Mach numbers 0.50–0.52 for Tra.Source: §30.2 · p. 322, Figure 30.1 · p. 322, §30.1 · p. 322

30.3 Stability Criteria

  • Networks trained with m <= 4 consistently failed to achieve stable rollouts, while m >= 8 was sufficient across the tested Mach numbers.The generalization test Mach number did not change this result.Source: §30.3 · p. 323
  • Residual versus direct prediction, an eightfold reduction in training data, and alternative backbones including dilated ResNets had little effect, though non-dilated ResNets were mostly unstable.The reduced dataset retained complete vortex-shedding periods; the non-dilated ResNets’ limited receptive field was identified as a likely issue.Source: §30.3 · p. 323, §30.5 · p. 325

30.4 Batch Size vs Rollout

  • Longer training rollouts paired with a batch size that uses available GPU memory improve stability more efficiently than smaller batches.Small batches can help stabilize networks, but they do not eliminate the need for unrolled training; larger batches also increase training time.Source: §30.1 · p. 321, §30.4 · p. 324, Figure 30.6 · p. 325

30.5 Summary

  • Unconditionally stable long-rollout predictions are achievable with suitable training; without diffusion, long training rollouts and sufficiently large networks are key requirements.The summary also favors longer rollouts over smaller batches when comparing stability and training time; suitable hyperparameters and enough training data are part of the learning setup.Source: UNCONDITIONAL STABILITY · p. 321, §30.5 · p. 325

31.1 Diffusion Graph Net (DGN)

  • On unstructured meshes, graph diffusion can learn converged-state distributions from short trajectories and incomplete per-system data, then sample states without simulating transient warm-up.The model is conditioned on each system’s mesh, boundary conditions, and physical parameters, and shared patterns across examples support interpolation across conditions.Source: GRAPH-BASED DIFFUSION MODELS · p. 327, §31.1 · pp. 327–328

31.2 Diffusion on Graphs

  • DGN uses a multi-scale graph U-Net processor to address message passing’s limited reach for global phenomena, applying message passing on the original and coarsened graphs.The approach is demonstrated with diffusion on mesh nodes, conditioned on graph features and the diffusion step.Source: §31.2 · pp. 328–329, Figure 31.2 · p. 329

31.3 Diffusion in Latent Space

  • LDGN models mid- to large-scale patterns in a VGAE latent space while the VGAE captures high-frequency detail, reducing sampling costs by operating on a smaller graph.The VGAE decoder also removes residual noise from generated samples.Source: §31.3 · pp. 329–330
  • The LDGN encodes states into a Gaussian latent space, learns its distribution conditioned on condition-encoder outputs, then decodes generated latents back to physical features.The condition encoder is needed at inference because the state-dependent encoder cannot be used then; conditions defined only on the coarse graph lack fine-grained details.Source: §31.3 · pp. 330–331

31.4 Turbulent Flows around Wings in 3D

  • The wing experiments model pressure distributions in 3D turbulent flow using surface meshes of about 7,000 nodes rather than regular grids requiring over 105 cells, with only 250 post-equilibrium training states.Those states represent about 10% of the data needed for statistically stationary variance.Source: §31.4 · p. 331

31.5 Distributional accuracy

  • Despite trajectories too short to capture standard deviations, spatial correlations, or higher-order statistics, DGN—especially LDGN—learns complete distributions and generates samples for in- and out-of-distribution settings.Evaluation compares 2,500 target states with 3,000 predicted samples.Source: §31.5 · p. 331
  • 1.95 ± 0.89 (LDGN), 2.12 ± 0.90 (DGN), and 4.32 ± 0.86 (Gaussian mixture model) are the reported graph-wise Wasserstein-2 distances, with LDGN lowest.The graph-wise measure penalizes inaccurate spatial correlations, unlike the node-wise measure.Source: §31.5 · p. 331

31.6 Computational Performance

  • LDGN generated 3,000 samples in 49 minutes on 8 CPU threads or 2.43 minutes on one GPU, versus 2,989 minutes for the simulator’s transient and 2,500 equilibrium states.For one converged state, the reported speedup is four orders of magnitude on CPU and five on GPU; LDGN is 8× faster than DGN with about 55% more training time.Source: §31.6 · p. 332

DISTRIBUTIONAL ACCURACY OF DIFFUSION GRAPH NETS

  • Diffusion training combines information across initial conditions and geometries to learn complete statistics, a capability distinguished from other probabilistic learning methods.The ellipse notebook evaluates pretrained models on pressure samples from the equilibrium distribution, not time evolution from an initial condition.Source: DISTRIBUTIONAL ACCURACY OF DIFFUSION… · p. 333

32.2 Sample-wise Accuracy

32.1 Implementation

  • On the ellipse task, DGN outperforms probabilistic baselines; flow-matching DGN and LDGN variants are included to assess faster inference.The baselines include a Gaussian Mixture Graph Net and a Variational Graph Autoencoder.Source: §32.1 · pp. 334–335

32.2 Sample-wise Accuracy

  • VGAE reached R2 = 0.9994 at t=36 on a single sample, but its apparent sample-wise quality does not reflect distributional accuracy.Across samples, VGAE has mode-collapse issues, which become more severe in more complex scenarios.Source: §32.2 · pp. 339–340, §32.3 · p. 346
  • The regular DGN can be slightly noisy in single outputs, while latent DGN performs better; both produce accurate pressure distributions.In distributional evaluation, regular DGN's noise averages out and can even surpass LDGN.Source: §32.2 · p. 340, §32.3 · p. 346
  • Flow matching uses one tenth the steps (5x faster) at the same sample-wise quality as denoising, but Flow-Matching DGN had R2 of std: 0.4237.Its Wasserstein-2 distance 1d was 0.0235 and Wasserstein-2 distance nd was 0.2256, compared with DGN's 0.0116 and 0.1536.Source: §32.2 · p. 340, §32.3 · p. 344

32.3 Evaluating Distributional Accuracy

  • DGN scored R2 of mean: 0.9989, R2 of std: 0.9814, Wasserstein-2 distance 1d: 0.0116, and Wasserstein-2 distance nd: 0.1536.The ground-truth distribution is bi-modal, and the DGN methods faithfully reproduce it.Source: §32.3 · pp. 344, 346
  • Bayesian Graph Net and Gaussian Mixture Graph Net produce problematic distributions, while VGAE exhibits mode collapse.Bayesian Graph Net had R2 of std: -0.6514 and Wasserstein-2 distance nd: 0.5987; Gaussian Mixture Graph Net had R2 of std: 0.1079 and Wasserstein-2 distance nd: 0.7174.Source: §32.3 · pp. 344–346

THIRTYTHREE

The section examines probabilistic methods and diffusion, PPO-based reinforcement learning, and differentiable physics for Burgers’ equation, including comparisons of trajectory reconstruction, generated forces, and convergence. It then discusses optimization under physical scaling, contrasting quasi-Newton and inverse-gradient updates with inverse-simulator methods and developing local inverse simulators and their theoretical behavior.

DISCUSSION OF PROBABILISTIC LEARNING

  • Diffusion methods enable learning and inference over solution distributions, but offer no accuracy advantage in deterministic settings and slightly raise inference cost, aside from long-term stability.For ambiguous problems, the text recommends flow matching; for reliable input-output pairs, it recommends simpler deterministic training.Source: DISCUSSION OF PROBABILISTIC LEARNING · p. 347

INTRODUCTION TO REINFORCEMENT LEARNING

  • Reinforcement learning trains an agent through environment interactions: it takes actions, receives state observations and rewards, and adapts behavior to increase cumulative reward.A neural-network policy selects actions from observations; the text divides RL algorithms into policy-gradient and value-based methods.Source: INTRODUCTION TO REINFORCEMENT LEARNING · p. 351

34.2 Proximal policy optimization

  • PPO is an actor-critic method chosen for controlling Burgers’ equation; it restricts how individual value estimates change the actor’s behavior to support stable learning, especially in continuous action spaces.PPO-clip [SWD+17] limits behavior changes per update using the previous network state.Source: §34.1 · p. 352, §34.2 · p. 352, §34.2.1 · p. 352

34.2.2 Actor

  • The actor samples actions around its predicted mean during training, while the critic predicts cumulative reward and uses Generalized Advantage Estimation [SML+15] to assess action value relative to an average.The training-time standard deviation decreases over training; inference evaluates the mean. GAE discounts summed advantage estimates using γ and λ, typically below one.Source: §34.2.2 · p. 353, §34.2.3 · p. 353

34.3 Application to inverse problems

  • For Burgers’ equation inverse control, PPO selects continuous force actions without solver gradients, using its value estimator to capture long-term effects and rewards that penalize force and final-state error.The target is to predict forces producing smooth transitions between states; the reward includes a per-step squared-force penalty and a penalty proportional to final L2 distance from the target.Source: §34.3 · p. 354, LEARNING · p. 357, §35.1 · p. 357

34.4 Implementation

  • The implementation uses stable_baselines3 PPO with ΦFlow 1.5.1, distinct actor and critic architectures, and a BurgersTraining class that manages training and evaluation against differentiable-physics training.The actor architecture follows a variant from Holl et al.; the critic uses convolutions, max pooling, and a final fully connected layer.Source: §34.4 · p. 355, §35.2 · p. 357

35.3 Data generation

  • The example trains on 800 of 1,000 simulated cases and reserves 100 each for validation and testing; vectorized environments collect parallel trajectories using physical and goal states as observations.The action space supplies one force value per velocity-field cell.Source: §34.4 · p. 354, §35.3 · p. 358

35.4 Training via reinforcement learning

  • The RL training setup runs PPO for 500 rollouts with 10 parallel environments, and its critic estimates rewards to update the actor that predicts control forces.The implementation periodically evaluates the agent using a predefined trajectory set to compare progress with the differentiable-physics method.Source: §34.4 · p. 355, §35.4 · p. 359
  • The trainer wraps PPO and a BurgersEnv, generates random initial states with ground-truth simulations, and evaluates performance in a separate validation environment.It creates a directory for models, logs, and hyperparameters, and loads an existing agent when the specified experiment folder exists.Source: §35.4 · p. 359

35.4.1 Gym environment

  • The Gym interface defines observation and action spaces and reset, step, and render methods; vectorized environments run multiple trajectories concurrently to improve resource use.The vectorized step operation is divided into asynchronous and waiting phases, allowing environment instances to run on separate threads.Source: §35.4.1 · pp. 359–360

35.4.2 Physics simulation

  • The Burgers environment uses current and target velocity fields plus time as observations, and applies agent actions as forces while advancing the simulation.Rewards penalize squared force at each step and subtract a scaled final L2 distance to the target; rewards are normalized using running estimates.Source: §35.4.2 · p. 360

35.4.3 Neural network setup

  • The RL setup uses PPO with a U-Net actor and a convolutional-and-pooling critic, and the demonstrated run fine-tunes an agent already trained for 3,500 iterations.The additional 500 rollouts typically take around two hours; training from scratch is possible but would take longer.Source: §35.4.3 · p. 360

35.5 RL evaluation

  • The RL agent reconstructs trajectories fairly well, but its results are noticeably less smooth than ground truth.Source: §35.5 · p. 362

35.6 Differentiable physics training

  • The differentiable-physics model reconstructs trajectories closely, with results that appear less noisy than those from the RL agent.Source: §35.6 · p. 366

35.7 Comparison between RL and DP

  • Generated force is used to compare RL and differentiable physics because minimizing it discourages the trivial strategy of applying a huge force at the final step.The authors describe low force use as an indicator that a method has learned the Burgers equation’s dynamics.Source: §35.7 · p. 366

35.7.1 Trajectory comparison

  • The trajectory comparison generates test-set trajectories with both methods and displays them alongside ground-truth simulations and natural, uncontrolled evolution.Source: §35.7.1 · p. 366
  • Both DP and RL approximate the ground truth, but DP generates less noisy trajectories.This is the comparison reported for the plotted trajectories.Source: §35.7.1 · p. 367

35.7.2 Comparison of exerted forces

  • DP applies slightly lower forces than RL, reaches target states on par with it, and generates forces closer to ground-truth values.The comparison identifies target-state accuracy as the main performance measure, while the force-value plots show DP predictions closer to ground truth.Source: §35.7.2 · pp. 368–369

35.8 Training progress comparison

  • PPO produces higher forces, slightly inferior control, and significantly slower convergence than DP in wall time and training iterations.The discussion attributes DP’s faster convergence to gradients from its tightly coupled solver, while RL relies on on-policy data collection and brute-force exploration.Source: §35.8 · pp. 370–371

35.9 Next steps

  • The proposed next steps are to test hyperparameter effects, compare the approaches at different field resolutions, and assess generalization across environment parameters.The suggested environment parameters include viscosity and dt; larger resolutions are expected to make physical dynamics harder to control.Source: §35.9 · p. 371

SCALE-INVARIANCE AND INVERSION

  • The chapter argues regular gradients can be suboptimal for physical learning, citing DP-gradient scaling issues [HKT22] and multimodal solutions while motivating physics-aware alternatives.It identifies missing inversion in prior GD and Adam runs and introduces analytical full-inversion and numerical half-inversion schemes.Source: SCALE-INVARIANCE AND INVERSION · p. 375

36.1 The crux of the matter

  • Physical operators scale dimensions differently, distorting ordinary gradients; inversion should preserve descent while making optimization scale-invariant across parametrizations.In the toy example, changing the scale parameter changes optimization trajectories without scale-invariant updates.Source: §36.1 · p. 376

36.2 Traditional optimization methods

  • The chapter compares gradient descent, Quasi-Newton methods, and inverse Jacobians, discussing their theoretical pros and cons for multivariate optimization.It also asks why classical optimization algorithms, despite some advantages, are not widely used for neural-network training.Source: §36.2 · p. 377

36.3 Gradient descent

  • Gradient descent mis-scales updates across parameters with different units and sensitivities, causing exploding or vanishing steps and slow convergence near optima.Normalization is unavailable within a simulation without destroying its state, and choosing a learning rate is difficult across time steps or varying output magnitudes.Source: §36.3 · pp. 377–378

36.4 Quasi-Newton methods

  • Quasi-Newton methods produce correctly dimensioned parameter updates with a dimensionless step size, counteract scaling by 1/α², and take larger steps in flatter loss regions.The larger steps arise because inverse-Hessian eigenvalues increase as the loss landscape becomes flatter.Source: §36.4 · pp. 378–379

Consistency in function compositions

  • Quasi-Newton updates still depend on earlier functions in a composition, while Hessian computation grows quadratically with parameter count and inversion can destabilize near-singular or nondifferentiable cases.The text also notes that many ReLU networks have zero second-order derivatives.Source: Consistency in function compositions · p. 379

36.5 Inverse gradients

  • Inverse gradients automatically scale updates to parameter units and function sensitivity, can improve convergence near optima, and allow intermediate-space changes to be estimated during backpropagation.Their compositional consistency holds up to first order.Source: §36.5 · p. 380

36.5 Inverse gradients; 36.6 Inverse simulators

  • Inverse gradients require an invertible square Jacobian, whereas a local inverse simulator can iteratively update from a nearby guess and exactly realize output-space steps despite nonlinear simulation.A full inverse could make a single inverse problem trivial, but iterative updates are considered for collections of targets or non-obvious solutions.Source: §36.5 · p. 380, §36.6 · p. 381

36.7 Summary

  • Inverse-simulator updates combine correct units, sensitivity handling, compositional consistency, and convergence near optima, but require a local inverse solver and can capture nonlinear effects beyond Newton and inverse gradients.The summary presents this as an ideal setting for improved, scale-invariant updates, while noting that constructing and discretizing the inverse solver has a cost.Source: §36.7 · p. 381

36.8.1 Fundamental theorem of calculus

  • Using the fundamental theorem of calculus, the inverse-simulator step averages local gradients along the path between states, effectively smoothing the optimization landscape to account for nonlinearities.The formulation generalizes to higher dimensions using a path integral along a differentiable path.Source: §36.8.1 · p. 382

36.8.2 Global and local inverse simulators; 36.8.3 Time reversal

  • A local inverse can exist even without a global inverse, selecting a nearby solution; time reversal may provide an inverse simulator, but accumulated numerical errors or insufficient simulator accuracy can limit it.For differentiable functions, a nonsingular Jacobian guarantees a local inverse; reversing time is a starting point unless information is lost in practice.Source: §36.8.2 · pp. 382–383, §36.8.3 · p. 383

36.8.4 Integrating a loss function

  • With a loss function, inverse-simulator updates match inverse gradients to first order while retaining nonlinear terms; accuracy depends on inverse fidelity, and a Newton step can determine the output-space step.The text notes that a regular gradient can serve as a fallback in the worst case.Source: §36.8.4 · p. 383

THIRTYSEVEN

This section compares gradient-based, Newton, and inverse-simulator updates, then develops scale-invariant physics training (SIP), including its loss, iterative extensions, and convergence behavior. It discusses SIP’s application to ill-conditioned problems, its dependence on the solver, network optimization, and task objective, and its relation to supervised training when solutions are multimodal.

37.1 Problem formulation

  • The toy problem squares the second input component and minimizes the resulting loss from x = [3, 3], with [0, 0] as the minimizer.The example uses a two-dimensional input and output with a scalar loss.Source: §37.1 · p. 385

37.2 3 Spaces

  • The example tracks three spaces: a forward pass maps x to simulator output y and then loss L, while optimization maps changes in L back to positions in x.The figure emphasizes x-space accuracy while also tracking optimization progress in y and L.Source: §37.2 · p. 385, Figure 37.1 · p. 386

37.3 Implementation

  • Using JAX to differentiate the composite loss gives gradient [6, 108] at the initial position, with the 108 component dominating the direction.The example defines a composite function for the loss and simulator because many JAX operations require a single native Python function.Source: §37.3 · pp. 386–387

37.4 Gradient descent

  • The last listed gradient-descent position is [2.451, 0.962], still 2.63 from the origin, as updates curve toward the origin but do not get very far.Source: §37.4 · pp. 388–389

37.5 Newton

  • Newton’s method follows a more diagonal trajectory than gradient descent and slows down less in this example, where its Hessian can be evaluated reliably.After 10 steps, Newton ends 0.925 from the origin, compared with 2.63 for gradient descent.Source: §37.5 · pp. 390–391

37.6 Inverse simulators

  • PG’s final position was [0.084, 0.504], 0.51 from the origin, compared with Newton’s 0.925 and gradient descent’s 2.63; its trajectory was better aligned with the ideal diagonal.The inverse function for y supplied higher-order terms than Newton’s second-order approximation, improving optimization scale-invariance and direction.Source: §37.5 · p. 390, §37.6 · pp. 392–393

37.7 y Space

  • For PG, optimization proceeds from L to y and then through the inverse simulator to x, so the method directly records its trajectory in y-space.The example stores that trajectory in the historyPGy list.Source: §37.7 · p. 394
  • In y space, the inverse simulator advances straight diagonally to the origin, while GD is suboptimal and Newton remains curved; both conventional methods leave intermediate states unknown until the full chain is evaluated.Source: §37.7 · pp. 394–395

37.8 Conclusions

  • In the simple example, GD gets stuck with unbalanced updates, Newton improves but remains nonoptimal, and inverse-simulator higher-order information outperforms both even when applied partially.Source: §37.8 · p. 396

37.9 Approximate inversions

  • 0.000001 MAE separates analytic PG from BFGS approximate inversion, which reproduces regular PG behavior after numerically finding a local inverse.Source: §37.9 · pp. 397–398

37.10 Next steps

  • The suggested next steps are to try more complex loss functions and compare the existing trajectories with those from AdaGrad, RMSProp, or Adam.Source: §37.10 · p. 398

38.1 NN training

38.2 Loss functions

  • For SIP, a Newton step in y provides the loss update without a full-system Hessian, while a proxy x-space L2 loss connects the inverse-physics and network graphs rather than serving as a traditional supervised loss.Source: §38.2 · p. 400

38.3 Iterations and time dependence

  • SIP extends from single predictions to control and simulation-correction tasks by repeating the loss, inverse-physics, and network updates across time, though repeated backpropagation makes training more difficult.Source: §38.3 · p. 401

38.4 SIP training in action

  • With ξ=1, the well-conditioned problem converges for all tested network optimizers, with Adam having a slight advantage; the comparison includes SIP with a saddle-free Newton solver.Source: §38.4 · p. 401, Figure 38.3 · p. 402
  • At 𝜉=32, only SIP and Adam significantly optimize the badly conditioned case; SIP’s Hessian-based updates retain relative accuracy across a wide range of 𝜉.Traditional optimizers lose accuracy as gradients scale with (1/𝜉, 𝜉), while SIP aligns updates with the flat direction; Adam also worsens as 𝜉 increases.Source: §38.4 · pp. 402–403
  • SIP training yields the best results across the varying optimization problems, while Adam deteriorates by more than an order of magnitude when parameters are coupled.SIP handles coupled parameters with a full-rank Hessian; Adam’s diagonal approximation lacks off-diagonal terms.Source: Figure 38.4 · p. 402, §38.4 · p. 403

38.5 Discussion of SIP Training

  • SIP’s scale-invariance can find solutions exponentially faster than other learning methods for many physics problems, with low computational cost.It provably converges with enough network updates Δ𝜃 per solver evaluation and can converge after a single update across a wide range of physics experiments.Source: §38.5 · p. 403

38.5.1 Limitations

  • SIP requires an approximately scale-invariant physics solver, and first-order optimizers for Δ𝜃 can still impair performance when network outputs have very different scales.Low-dimensional spaces may use Newton’s method; high-dimensional inversion may require numerical methods or domain-specific knowledge, while NN inversion remains a goal.Source: §38.5.1 · p. 403
  • SIP’s equal weighting of examples can distort loss-based priorities when curvature varies, so traditional methods may perform better when x-space accuracy is unimportant.This can matter in control tasks or when regions with very small or large curvature distort sample importance.Source: §38.5.1 · p. 403

38.5.2 Similarities to supervised training

  • SIP resembles supervised training by computing reliable inversion updates on the fly, avoiding the suboptimal solutions that precomputed datasets can learn when mappings are non-unique.Precomputed data cannot represent multimodality in such mappings; differentiable-physics gradients can also remain improperly inverted and cause vanishing or exploding gradients.Source: §38.5.2 · p. 404

THIRTYNINE

The section explores optimization methods for physics-based inversion and neural-network training through heat-conduction and linear-oscillator examples, comparing SIP, HIG, and conventional optimizers under different conditioning. The examples report differences in reconstruction and training loss, and identify test-data evaluation and broader physics settings as next steps.

39.1 Problem Statement

  • Heat inversion becomes increasingly unstable as t·ν grows because diffusion evens out initially distinct heat levels and obscures the original information.Source: §39.1 · p. 405

39.2 Implementation

  • The example tests a challenging diffusion level of t·ν=8 on a 64x64-cell domain, where most detail is diffused and large-scale structure remains.Source: §39.2 · p. 405

39.3 Data generation

  • Training inputs are generated on the fly by placing 4–10 random hot rectangles in the domain, and the network must reconstruct them from their diffused states.Source: §39.3 · p. 406

39.4 Differentiable physics and gradient descent

  • Differentiable-physics gradients undergo further diffusion, so gradient descent converges slowly and struggles to recover high-frequency details even though the information is not fundamentally missing.Source: §39.4 · p. 407

39.5 Stable SIP gradients

  • Probabilistic damping stabilizes inverse diffusion by limiting exponential amplification of noisy high frequencies while retaining as much high-frequency gradient information as the noise allows.The method models observations as signal plus noise and uses a Bayesian signal probability to dampen the inverse-physics amplification.Source: §39.5 · pp. 407–408

39.6 Neural network and loss function

  • SIP shows significantly improved convergence on reconstructing the initial field over DP+Adam, due to a stronger physics signal from the inverse simulator’s higher-order step.Both variants use the same backpropagation-based update for network weights.Source: §39.8 · p. 411
  • The SIP reconstruction is sharper and has fewer halos than the GD reconstruction, because GD diffuses the backpropagated gradient rather than working against it.SIP improvements are less pronounced in the physical space, which is less important than the x reconstruction for this inverse problem.Source: §39.8 · p. 412

HALF-INVERSE GRADIENTS

  • HIGs jointly invert the neural-network and physics components without an analytical inverse solver, but require an SVD of a large Jacobian and use first-order information.They use the full Jacobian, unlike regular gradients, and typically outperform GD and Adam significantly.Source: HALF-INVERSE GRADIENTS · p. 415

40.1 Derivation

  • HIG updates use a truncated half-inverse computed by SVD; partial inversion regularizes the update and reduces the risk that small singular values cause excessive contributions or gradient explosions.Singular values below a threshold are discarded rather than clamped, because their associated singular vectors may be unreliable.Source: §40.1 · p. 416

40.2 Constructing the Jacobian

  • HIGs concatenate per-sample Jacobians across a minibatch to find an L2-optimal joint update direction rather than averaging directions, and often benefit from larger minibatches.Computing the update entails an SVD of the combined Jacobian, truncating and half-inverting its singular values, then reassembling the update direction.Source: §40.2 · pp. 416–417

40.4 Well-conditioned

  • In the well-conditioned case, Adam, Gauss–Newton, and HIG all reduce loss to around 10^-2, avoid neuron saturation, and reach the target output.Gauss–Newton and HIG make larger initial jumps than Adam, whose momentum reduces the jumpiness.Source: §40.4 · pp. 417–418, Figure 40.2 · p. 418

40.5 Ill-conditioned

  • At λ=0.01, Adam and Gauss–Newton stall near 0.2; Adam struggles with y2 scaling, while Gauss–Newton can explode and saturate neurons, leaving both short of the target.The ill-conditioned case contrasts with loss levels of 0.01 and better when λ=1; saturated neurons can produce effectively zero future gradients.Source: §40.5 · pp. 418–419, Figure 40.3 · p. 419

COUPLED OSCILLATORS WITH HALF-INVERSE GRADIENTS

  • The practical example compares Adam, Scale-Invariant Physics, and HIGs on learning a control function that returns a two-mass coupled oscillator to its initial state after 24 fourth-order Runge–Kutta steps.The control is learned with a neural network to steer the oscillator dynamics toward the desired position and velocity at the chosen time.Source: COUPLED OSCILLATORS WITH HALF-INVERSE… · p. 421, §41.1 · p. 421

41.3 Coupled linear oscillator simulation

  • The coupled oscillator solver advances states in batches with a fourth-order Runge–Kutta step over the simulation time steps.The solver repeatedly calls the batched RK4 routine with the current control input.Source: §41.3 · p. 423

41.4 Training setup

  • The network has four dense layers with three 20-neuron tanh hidden layers, and it is trained with L2 loss on 4k random initial states as return-to-start targets.The target states equal the initial states.Source: §41.4 · p. 424

41.5 Training

  • Training uses method-specific heuristic settings: Adam represents gradient descent, SIP uses Gauss–Newton physics inversion, and HIG uses SGD with a Jacobian half-inversion.The physics inversion is exact for the quadratic physical optimization landscape; HIG's half-inverse is constructed using an SVD and square-rooted singular values, and training is time-limited.Source: §41.5 · pp. 424–425
  • 1.1857609933940694e-06 training loss was reached by epoch 4, down from 1.4401766061782837 at epoch 0.The reported losses for epochs 1–3 were 5.44398972124327e-05, 1.064436037268024e-05, and 3.163525434501935e-06.Source: §41.5 · p. 429
  • Adam initializes and runs iterations faster, but its rough Hessian approximation stagnates at high loss; SIP does not outperform Adam in this simple-physics case.The runtime comparison includes TensorFlow initialization, which is more involved for HIG and SIP than for Adam.Source: §41.6 · pp. 430–431
  • HIG reaches loss accuracy around four orders of magnitude better than Adam and SIP, converging quickly despite slower iterations.Its half-inversion produces a strong update; Adam stagnates at high loss, while SIP may overtake Adam if run longer but can encounter numerical issues with full inversion.Source: §41.6 · pp. 431–432

41.7 Next steps

  • The notebook evaluates training performance only, so a proposed next step is to allocate test samples and compare all three methods on test data.The text says HIG is claimed to work on real test data, but this notebook has not evaluated that claim.Source: §41.7 · p. 432
  • Further tests can vary oscillator count and time span or add a nonlinear force, which requires a new inverse solver for SIP.Source: §41.7 · p. 432

FORTYTWO

This section surveys methods for physics-based deep learning, from inversion-based training and latent-space simulation to mesh- and particle-based approaches, continuous convolutions for liquid simulation, and GANs for generating physical solutions. It considers how learned representations and methods adapted to physical structure address challenges in scaling, accuracy, and temporal or spatial complexity, alongside GAN training challenges, applications, and future directions.

42.1 Addressing scaling issues

  • HIGs jointly invert neural-network and physics components for scale-invariant training, while SIPs use custom inverse solvers to make the physics component scale-invariant.HIGs use Jacobians and all mini-batch samples to compute an optimal first-order direction; SIPs limit scale invariance to the physics component.Source: §42.1 · p. 433, §42.3 · p. 434

42.2 Computational Resources

  • Improved-gradient methods can cost more to train and require memory-intensive SVDs, but their more accurate reusable models can pay off in repeated downstream use.The text says comparable accuracy from regular Adam and DP-based training may be infeasible, while improved accuracy can save resources in repeated evaluations.Source: §42.2 · p. 433, §42.3 · p. 434

42.3 Summary

  • In PDE-based learning, inversion can provide more useful optimization information than regular neural-network gradients, often improving convergence and model performance.Even SIPs, which invert only the physics component, can use higher-order information through custom inverse solvers.Source: §42.3 · p. 434

MODEL REDUCTION AND TIME SERIES

  • ROMs reduce high-dimensional simulation states to latent variables whose evolution is predicted and decoded; open-source code supports end-to-end simulation and encoding–decoding.Time-dependent 3D discretizations can require O(n^4) samples; latent dimension m must capture important solution structures while enabling efficient prediction.Source: MODEL REDUCTION AND TIME SERIES · p. 439, §44.4 · p. 441

44.1.1 Autoencoder variants

  • Autoencoders learn nonlinear latent representations that can achieve high accuracy with fewer degrees of freedom, while VAEs shape the latent space to support sampling.The encoder and decoder must be separable so either can operate on its own; VAEs typically target a normal latent distribution, though simulation prediction may matter more than easy sampling.Source: §44.1 · p. 440, §44.1.1 · p. 440

44.2 Time series

  • Latent-state prediction is recurrent, so training across multiple steps helps expose accumulated drift; models need adequate state history or recurrent memory.LSTMs, GRUs, and transformers are among the recurrent options, while fully connected prediction layers can grow quickly and favor small latent dimensions.Source: §44.2 · pp. 440–441

44.3 End-to-end training

  • End-to-end training can reduce overall inference error despite greater training cost, and deep learning enables nonlinear manifolds and complex evolution in latent space.Unrolling prediction over time trains the components together; the example also couples learned evolution with a numerically solved advection step.Source: §44.3 · p. 441

45.2 Unstructured meshes and graph neural networks

  • Structured grids simplify deep-learning implementations, while adaptive unstructured or meshless approaches support irregular configurations; particle methods can use continuous convolutions over neighborhoods.The particle-based method offers spatial adaptivity and a real convolution operator for learning physical relationships; meshless methods can recompute neighborhoods as connectivity changes.Source: §45.1 · p. 443, §45.2 · p. 444, §45.3 · p. 444

45.5 Learning the dynamics of liquids

  • The learned liquid solver achieves good accuracy with a small, efficient model that uses fewer weights and evaluates faster than GNN-based approaches.Because the trained solver is differentiable, it also supports gradient-based optimization of inputs such as viscosity. The practical implementation requires fast neighbor collection; source code includes training code for liquid dynamics.Source: §45.5 · p. 445, §45.6 · p. 446

46.2 Adversarial training

  • GANs train a discriminator to distinguish generated from real samples, then use its gradients to steer the generator; classification cross-entropy is maximum likelihood estimation.Training typically alternates discriminator and generator updates.Source: §46.2 · p. 448, §46.1 · p. 448

46.3 Regularization

  • GAN training is finicky because coupled optimization requires a fragile balance, and mode collapse can break training.L1 regularization or supervised generator pre-training can help stabilize training, with discriminator pre-training sometimes needed to maintain balance.Source: §46.3 · p. 448

46.5 Ambiguous solutions

  • For multimodal problems such as super-resolution, supervised learning can average valid solutions into an undesirable result, while GANs can avoid this averaging.Conditional GANs use an input such as parameters, boundary conditions, or a partial solution to generate an output; low-resolution inputs are one example.Source: §46.5 · p. 449, §46.4 · p. 449

46.6 Spatio-temporal super-resolution

  • Spatio-temporal self-supervision produces GAN results that closely match the reference solution by having the discriminator judge temporal evolution from reference-solution triplets.The cited work extends learned self-supervision to space-time fluid super-resolution.Source: §46.6 · pp. 449–450

46.7 Physical generative models

  • A cyclic GAN learned realistic Navier–Stokes solutions across varying physical parameters from data alone, without a differentiable physics solver.The challenging solution manifold includes buoyancy, vorticity, boundary conditions, and obstacle geometries; the figure describes extrapolation beyond training buoyancy values.Source: §46.7 · p. 450, Figure 46.4 · p. 450

46.8 Discussion

  • When a reasonable differentiable PDE model is available, differentiable-physics training is expected to outperform approximating the PDE with a GAN discriminator.GANs can still be attractive with black-box solvers that lack gradients; the discriminator can be discarded after generator training.Source: §46.8 · pp. 450–451

OUTLOOK

  • The outlook frames deep learning as a complement to numerical methods and points to chemical reaction, plasma, and weather and climate modeling as promising directions.It also identifies extracting human-readable insights from learned solution manifolds as a future challenge.Source: OUTLOOK · p. 455, §47.1 · p. 455

Related papers

Loading 2109.05237v4…