PaperlayerView exact revision on arXiv

Source-linked AI summary

Enforcing Statistical Constraints in Generative Adversarial Networks for Modeling Chaotic Dynamical Systems

Jin-Long Wu, Karthik Kashinath, Adrian Albert, Dragos Chirila, Prabhat, Heng Xiao

arXiv:1905.06841v1physics.comp-phphysics.flu-dynstat.ML

TL;DR

Standard GANs can struggle to capture training-data statistics for complex distributions. The paper constrains GANs using covariance information and reports up to 80% lower training cost to reach a solution, with similar or better quality.

  • Problem

    Standard GANs can be difficult to train and may converge to local minima, making it unlikely they capture all training-data statistics as distribution complexity increases.

    Source: §1 · p. 3

  • Method

    The approach quantifies differences between generated-sample and training-data covariance structures and incorporates them into a statistical constraint for GAN training.

    Source: §4 · p. 21

  • Results

    Up to 80% lower training cost was reported for the constrained GAN to reach a solution, with similar or even better quality than standard GANs.

    Source: §4 · p. 22

  • Takeaways & Limitations

    The authors conclude that statistical regularization leads to more robust training than standard GANs.

    Source: §4 · p. 21

    The reported result comes from experiments on an idealized system.

    Source: §3 · p. 10

Abstract

from arXiv · show

Simulating complex physical systems often involves solving partial differential equations (PDEs) with some closures due to the presence of multi-scale physics that cannot be fully resolved. Therefore, reliable and accurate closure models for unresolved physics remains an important requirement for many computational physics problems, e.g., turbulence simulation. Recently, several researchers have adopted generative adversarial networks (GANs), a novel paradigm of training machine learning models, to generate solutions of PDEs-governed complex systems without having to numerically solve these PDEs. However, GANs are known to be difficult in training and likely to converge to local minima, where the generated samples do not capture the true statistics of the training data. In this work, we present a statistical constrained generative adversarial network by enforcing constraints of covariance from the training data, which results in an improved machine-learning-based emulator to capture the statistics of the training data generated by solving fully resolved PDEs. We show that such a statistical regularization leads to better performance compared to standard GANs, measured by (1) the constrained model's ability to more faithfully emulate certain physical properties of the system and (2) the significantly reduced (by up to 80%) training time to reach the solution. We exemplify this approach on the Rayleigh-Benard convection, a turbulent flow system that is an idealized model of the Earth's atmosphere. With the growth of high-fidelity simulation databases of physical systems, this work suggests great potential for being an alternative to the explicit modeling of closures or parameterizations for unresolved physics, which are known to be a major source of uncertainty in simulating multi-scale physical systems, e.g., turbulence or Earth's climate.

1. Introduction

The introduction argues that standard GANs may miss important statistics of complex physical systems and presents statistical constraints as a way to improve their emulation. The work focuses on statistical constraints, while benefits of physical constraints are discussed separately.

1. Introduction

  • Unresolved multi-scale processes require closure models, which are a major source of simulation uncertainty partly because they neglect high-order statistics.Source: §1 · p. 2
  • As true-data complexity increases, standard GANs may converge to local minima and fail to capture all statistics or reproduce important physical properties.Although they can capture the training distribution at a global loss minimum, GANs are difficult to train and may miss properties when convergence is local.Source: §1 · p. 3
  • The paper introduces statistical constraints to help GAN emulators preserve high-order statistics in complex PDE-governed systems.The study focuses on statistical constraints; benefits of physical constraints are presented separately.Source: §1 · p. 4
  • The authors tested statistical constraints on three relatively simple canonical systems and discuss extensions to more complex systems, aiming to support reliable alternatives to closure models.The longer-term aim is to leverage high-fidelity simulation output to emulate unresolved processes.Source: §1 · p. 4

2. Methodology

The methodology evaluates covariance-constrained GAN training for preserving data statistics using turbulent-regime data from a high-resolution, two-dimensional Lattice Boltzmann simulation of Rayleigh–Bénard convection, presented as a useful test despite its limited physical scope.

2.1. Generative adversarial networks

  • Standard GAN training is a minimax game between generator and discriminator, but its global optimum is difficult to reach and may leave training-data statistics uncaptured.The paper notes that missing second-order velocity moments matter in PDE-governed systems because they correspond to Reynolds stress in turbulence modeling.Source: §2.1 · p. 5

2.2. Constrained generative adversarial networks

  • The constrained GAN adds a loss penalty for the distance between training-data and generated-sample covariance structures.The penalty explicitly estimates second-order moments, with generated-sample covariance estimated during each training iteration.Source: §2.2 · pp. 6–7
  • The Frobenius norm was the best compromise between computational cost and training stability among the covariance-distance measures tested.Source: §2.2 · p. 6

2.3. Lattice Boltzmann Simulation of Rayleigh-Bénard convection

  • RBC consists of a fluid slab between horizontal surfaces, with buoyancy-driven flow when the lower wall is heated and the upper wall cooled.Source: §2.3 · p. 7
  • RBC flow regimes depend on initial conditions and control parameters including Rayleigh number, Prandtl number, and the domain’s horizontal-to-vertical aspect ratio.Source: §2.3 · p. 8

2.3. Lattice Boltzmann Simulation of Rayleigh-B´enard convection

  • The RBC training data came from a two-dimensional fluid-dynamics simulation based on the Lattice Boltzmann method, with Pr = 0.71 and Ra = 2.5 × 10^8.Source: §2.3 · p. 8
  • Total kinetic energy and lower-wall Nusselt number were already stabilized at the start of the GAN training window, indicating that the training data belonged to the final turbulent regime.The model was initially run for 400 eddy turnover times, and the initial 320 were discarded for spin-up.Source: §2.3 · p. 8
  • The training-window mean lower-wall Nusselt number was ⟨Nu0⟩= 35.1137, close to ’s ⟨Nu0⟩= 34.1422 for a similar setup with Γ = 2.The authors note that the lower aspect ratio and periodic boundary conditions in artificially constrain the flow.Source: §2.3 · p. 8
  • The two-dimensional training data, and thus the GAN results, omit three-dimensional phenomena such as vortex stretching, but the authors consider 2D studies useful tests because they share much physics with 3D flows.A full three-dimensional case is left for future work.Source: §2.3 · pp. 8–9

3. Results

The results show that statistical constraints improve GAN stability and fidelity to training-data statistics across both Rayleigh–Bénard cases, while reducing training cost by up to 80%. At high Rayleigh number, a 20-epoch constrained GAN captured important statistics better than standard GANs, including a 100-epoch TKE result.

3. Results

  • The covariance constraint better captured training-data statistics and supported more stable training, although fine-tuned standard GANs could achieve similar performance.The authors note that tuning depends heavily on user experience and may be infeasible for complex systems.Source: §3 · p. 10
  • Up to 80% lower computational cost in model training accompanied higher-quality results for the constrained GAN versus the unconstrained model.The authors attribute the reduced cost to restricting training to a smaller space of allowable solutions.Source: §3 · p. 10

3.1. 2-D Gaussian Process

  • In the 2-D Gaussian-process experiment, the constrained GAN better matched covariance and correlation symmetry, while standard-GAN covariance was noticeably asymmetric.The comparison is shown for covariance around the center point and correlation profiles along two diagonals.Source: §3.1 · p. 11, Figure 4 · p. 11, Figure 5 · p. 12
  • The constrained GAN generally yielded smaller covariance-distance metrics across GAN architectures and better agreement with training-data skewness and kurtosis.Skewness and kurtosis were normalized against multivariate-Gaussian benchmark values, for which the ideal value is one.Source: §3.1 · p. 12, Figure 7 · p. 13

3.2. Rayleigh-B´enard convection at a low Rayleigh number

  • At low Rayleigh number, the constrained GAN was more stable to training-parameter changes; the standard GAN’s TKE pattern changed significantly, while the constrained result changed less.The constrained result was noisier at the larger learning rate, mainly because that rate introduced more training noise.Source: Figure 9 · p. 15, §3.2 · p. 15
  • At low Rayleigh number, constrained-GAN TKE-field MSEs were generally smaller than standard-GAN MSEs; at a larger learning rate, standard GAN did not converge and had noticeably larger MSE.At 100 epochs, performances were comparable across penalty coefficients.Source: §3.2 · p. 15, Figure 10 · p. 16
  • At lr = 0.002, the statistical constrained GAN’s spectrum showed marginal improvement and better agreement; it showed much better performance than standard GAN, whose energy spectrum disagreed with training data across the whole range of wave numbers.Source: §3.2 · p. 16

3.3. Rayleigh-B´enard convection at a high Rayleigh number

  • At the high-Rayleigh-number center and corner points, constrained-GAN velocity distributions at 20 epochs outperformed standard GAN at 20 epochs and were comparable to standard GAN at 100.Standard GAN distributions at 20 epochs were noticeably biased.Source: §3.3 · pp. 16–17, Figure 12 · p. 18, Figure 13 · p. 18
  • At 20 epochs, the constrained GAN better preserved high-Rayleigh-number TKE statistics than the standard GAN at 100 epochs.The authors conclude that the constrained GAN better preserves statistics such as the mean and second-order moment.Source: Figure 15 · p. 20, §3.3 · p. 20
  • At high Rayleigh number, the model at 20 epochs successfully captured the training-data energy spectrum for most wave numbers, except very large wave numbers (∼102).At 20 epochs, the standard GAN overestimated the energy spectrum across the whole range of wave numbers.Source: §3.3 · pp. 20–21, Figure 16 · p. 21

4. Conclusion

GANs can emulate PDE-governed systems but are difficult to train; this work uses covariance-based statistical constraints to improve training robustness.

4. Conclusion

  • The method penalizes differences between generated-sample and training-data covariance structures in the generator’s original loss function.Source: §4 · p. 21
  • The constrained GAN was evaluated against standard GANs on Gaussian random fields and Rayleigh–Bénard convection.Source: §4 · p. 21
  • Up to 80% lower training cost was achieved by constrained GANs while reaching similar or better quality, with more robust training than standard GANs.Carefully tuned standard GANs could achieve similar quality.Source: §4 · pp. 21–22
  • With growing high-fidelity simulation databases, the approach may offer an alternative to explicit closure or parameterization modeling by preserving high-order statistics in chaotic physical systems.Source: §4 · p. 22

Related papers

Loading 1905.06841v1…