PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations

Liu Yang, Dongkun Zhang, George Em Karniadakis

arXiv:1811.02033v1stat.MLcs.LGmath.APmath.NA

TL;DR

The paper tackles forward, inverse, and mixed SDE problems from limited scattered measurements, where known physics and stochastic fields must be handled together. It introduces PI-GANs that encode SDE structure into GAN generators and use WGAN-GP training. The method produced good stochastic-process approximations and solved demonstrated SDE problems up to 30 dimensions.

  • Problem

    Limited scattered measurements must support inference of stochastic solutions and unknown equation terms in forward, inverse, and mixed physical problems.

  • Method

    PI-GANs combine feed-forward DNN generators with an SDE-induced neural network and WGAN-GP losses to approximate joint distributions of stochastic processes.

  • Results

    PI-GANs solved demonstrated SDE problems up to 30 dimensions, with generated stochastic quantities showing good agreement with benchmarks and reference values.

  • Takeaways & Limitations

    The framework solves forward, inverse, and mixed stochastic problems without changing its basic formulation, using one or multiple discriminators according to the data.

  • Takeaways & Limitations

    PI-GANs have higher computational cost than physics-informed neural networks, and limited-data experiments detected overfitting in both discriminators and generators.

Abstract

from arXiv · show

We developed a new class of physics-informed generative adversarial networks (PI-GANs) to solve in a unified manner forward, inverse and mixed stochastic problems based on a limited number of scattered measurements. Unlike standard GANs relying only on data for training, here we encoded into the architecture of GANs the governing physical laws in the form of stochastic differential equations (SDEs) using automatic differentiation. In particular, we applied Wasserstein GANs with gradient penalty (WGAN-GP) for its enhanced stability compared to vanilla GANs. We first tested WGAN-GP in approximating Gaussian processes of different correlation lengths based on data realizations collected from simultaneous reads at sparsely placed sensors. We obtained good approximation of the generated stochastic processes to the target ones even for a mismatch between the input noise dimensionality and the effective dimensionality of the target stochastic processes. We also studied the overfitting issue for both the discriminator and generator, and we found that overfitting occurs also in the generator in addition to the discriminator as previously reported. Subsequently, we considered the solution of elliptic SDEs requiring approximations of three stochastic processes, namely the solution, the forcing, and the diffusion coefficient. We used three generators for the PI-GANs, two of them were feed forward deep neural networks (DNNs) while the other one was the neural network induced by the SDE. Depending on the data, we employed one or multiple feed forward DNNs as the discriminators in PI-GANs. Here, we have demonstrated the accuracy and effectiveness of PI-GANs in solving SDEs for up to 30 dimensions, but in principle, PI-GANs could tackle very high dimensional problems given more sensor data with low-polynomial growth in computational cost.

1. Introduction

The paper addresses stochastic physical problems with limited scattered measurements by embedding known SDE physics into GANs, integrating model structure with data for forward, inverse, and mixed inference.

  • GANs can learn probability distributions, but prior work had not explicitly encoded known physical laws into GANs for stochastic problems.
  • The mixed setting combines scattered measurements with boundary and initial conditions to infer missing PDE terms or parameters while recovering a stochastic solution.The paper represents stochastic solutions and fields with SDEs arising from stochastic excitation or uncertain material properties.
  • PI-GANs encode the SDE form into GAN architectures while learning unknown stochastic terms from data.The approach targets inference, system identification, and mixed cases with insufficient data for both system and state.
  • Most related data-driven engineering methods address deterministic systems, with relatively few published methods for SDEs.
  • The paper studies stochastic-process learning from limited realizations, WGAN-GP overfitting, and stochastic PDE solutions across problem types.

2. Problem setup

The problem setup uses scattered sensor snapshots of stochastic coefficients, forcing, solutions, and boundary data under a general SDE formulation, with sensor availability determining whether problems are forward, mixed, or inverse.

  • The governing model uses a general differential operator with stochastic coefficient k(x; ω), forcing f(x; ω), solution u(x; ω), and boundary conditions.The coefficient and forcing are random processes, so the solution depends on both.
  • Measurements are collected as simultaneous snapshots from scattered sensors for k(x; ω), u(x; ω), f(x; ω), and b(x; ω).Each snapshot corresponds to one random event and concatenates the readings into a vector.
  • Terms with no corresponding sensors are omitted from the measurement representation.
  • Reducing sensors on k(x; ω) while increasing sensors on u(x; ω) transitions the problem from forward to mixed and then inverse.The setup assumes a sufficient number of sensors for f(x; ω).
  • Multiple independently collected snapshot groups can use different sensor setups, combining newly collected data with previously collected readings.The framework indexes groups and snapshots separately to accommodate this setting.

3. A brief review of GANs and WGANs

The paper reviews GANs and WGANs, emphasizing WGAN-GP’s Wasserstein objective and gradient penalty for more stable distribution matching, including low-dimensional targets and mismatched input dimensions.

  • A GAN generator maps random input z from Qz in IR^m to samples in IR^d whose distribution Pg approximates the target distribution Pr.
  • GAN training is a two-player game in which the discriminator distinguishes Pr from Pg while the generator imitates Pr.
  • Vanilla GANs use a JS-divergence-related objective, but this divergence may provide no usable generator gradient when distributions lie on low-dimensional manifolds.This can make vanilla GAN training delicate and unstable.
  • WGAN-GP imposes the discriminator’s Lipschitz constraint with a gradient penalty rather than weight clipping, and training is more stable than weight-clipped WGANs.The penalty uses points sampled uniformly along lines joining real and generated samples.
  • WGANs replace the JS divergence with the Wasserstein-1 distance, which is continuous and almost everywhere differentiable under a mild constraint.The Wasserstein objective is defined through joint distributions with the target and generated distributions as marginals.
  • In four IR10 toy problems, WGAN-GP converged to distributions close to the targets even with one-dimensional or ten-dimensional Gaussian input noise.The tests included targets supported on hypercubes and curves, with generated supports becoming increasingly twisted or concentrated during training.

4. Methodology

PI-GANs combine feed-forward neural networks with SDE-induced networks to solve forward, inverse, and mixed stochastic problems from scattered sensor measurements. The framework encodes physical operators, incorporates snapshots through adversarial training, and supports multiple data groups and discriminators.

  • Network architecture: PI-GANs represent stochastic processes with feed-forward DNN generators and SDE-induced neural networks.The generators for the solution, forcing, and diffusion coefficient are constructed from feed-forward networks and operator-induced networks.
  • Physics encoding: Automatic differentiation applies the SDE operators to feed-forward networks, producing generators for the forcing and diffusion coefficient.The induced generators inherit parameters from the networks representing the solution and diffusion coefficient.
  • Architecture schematic: The schematic maps feed-forward networks, operator-induced networks, sensor snapshots, sampling procedures, and concatenation to distinct visual elements.The operator-induced networks correspond to Nx and Bx.
  • Adversarial training: Training compares real and generated sensor snapshots adversarially, after which generated sample paths provide stochastic statistics.For multiple snapshot groups, separate fake groups can be generated and supplied to group-specific discriminators.
  • Multiple data groups: Multiple discriminators can focus on separate snapshot groups while generators learn to deceive all discriminators simultaneously.For a single data group, the method uses M = 1; the generator-loss weights are set to 1 in the described setup, though their selection requires further study.
  • Problem types: The same framework solves forward, inverse, and mixed problems without explicitly distinguishing among them.The problem type varies with sensor coverage of the diffusion coefficient and solution, but the PI-GAN framework remains unchanged.

5. Numerical Results

The numerical experiments use smooth tanh-activated networks and fixed generator and discriminator architectures, with sensor placement and noise inputs specified for the test cases.

  • Network settings: Tanh activations are used because piecewise-linear functions are unsuitable when SDE solutions require high-order derivatives.The feed-forward generator networks have 4 hidden layers of width 128.
  • Network settings: Most discriminators match the generator depth and width, with narrower architectures used for selected experiment sections.The Section 5.1 discriminators use width 64, while one additional-group discriminator uses width 16.
  • Training settings: The standard setup uses Xavier weight initialization, zero biases, Gaussian generator noise, default WGAN-GP and Adam hyperparameters, and equidistant sensors.The shared settings include λ = 0.1, nd = 5, α = 0.0001, β1 = 0, and β2 = 0.9.

5.1. A pedagogical problem: approximating stochastic processes

WGAN-GP approximated Gaussian and constrained stochastic processes from sparse sensor snapshots, with accuracy shaped by correlation length, sensor count, and training data. The experiments also identified generator and discriminator overfitting behaviors.

  • Gaussian-process setup: WGAN-GP approximated Gaussian processes across correlation lengths using 1,000 snapshots, 6 or 11 sensors, and four-dimensional input noise.The supplementary shortest-correlation-length case used 10,000 snapshots.
  • Sensor and snapshot effects: Smaller correlation lengths widened the gap from reference solutions with 1,000 snapshots, but additional snapshots narrowed it.Smaller correlation lengths produced higher effective dimensionality and subtler local behavior.
  • Sensor and snapshot effects: 11 sensors produced better approximations than 6 sensors when sufficient training data were available.More sensors were needed to describe processes with small correlation length.
  • Spectral approximation: Four-dimensional input noise reproduced spectra of target processes with much higher effective dimensionality.The authors attribute this to folding and twisting of a low-dimensional manifold into a higher-dimensional region.
  • WGAN-GP versus vanilla GANs: WGAN-GP remained accurate for the fixed-boundary case, whereas vanilla GANs failed there; both methods performed well for the Gaussian-process case.Both methods used 10,000 snapshots collected from 11 sensors.
  • Overfitting issues: Generator overfitting occurred in random space, while discriminator overfitting biased W1-distance estimates; physical-space generator overfitting was not detected.Random-space overfitting biased generated snapshots toward training snapshots, whereas physical-space statistics matched reference values well.

5.2. Forward problem

The forward PI-GAN problem approximates the stochastic solution from sparse measurements while recovering the effects of stochastic forcing and diffusion. Its inferred statistics and covariance structure agree with reference solutions, with accuracy improving as input-noise dimension or training snapshots increase.

  • 1000 snapshots from sparsely placed sensors were used to infer the mean and standard deviation of the stochastic solution u(x; ω).The setup also approximated the independent stochastic processes k(x; ω) and f(x; ω).
  • The inferred errors were comparable with those from 1000 full trajectory sample paths despite using only sparse sensor measurements.The comparison concerns the mean and standard deviation of u(x; ω).
  • Increasing input-noise dimension reduced standard-deviation error while leaving mean error relatively unchanged.The authors attribute the improvement to higher-dimensional manifolds filling high-dimensional regions more efficiently.
  • With fixed input-noise dimension, increasing the number of training snapshots decreased the error.The tested snapshot counts were 300, 1000, and 3000.
  • The generated forcing spectra fit reference solutions, and the gap narrowed with higher input-noise dimension or more training snapshots.The spectra were used to assess whether the covariance structure of f(x; ω) was captured.
  • Generated k(x; ω) and f(x; ω) were weakly correlated, whereas generated k(x; ω) and u(x; ω) were strongly correlated, with coefficients close to reference values.The correlation coefficients were evaluated using the definitions described for C(k, f) and C(k, u).

5.3. Inverse and mixed problems

The inverse and mixed experiments use the same PI-GAN framework across sensor configurations that distribute information about the diffusion coefficient, forcing, and solution. Across all three problem types, errors were of the same order as errors from 1000 Monte Carlo sample paths.

  • The four sensor configurations range from 1 to 13 k-sensors and from 2 to 13 u-sensors, with 13 f-sensors in each case.The u-sensors include boundary measurements where specified.
  • Cases 1, 2, and 3 represent inverse or mixed problems, while case 4 is a forward problem using 1000 training snapshots.The cases differ in the allocation of k-, u-, and f-sensors.
  • Errors for inverse, mixed, and forward problems were in the same order of magnitude as errors from 1000 Monte Carlo sample paths.The comparison is reported for the relative errors of inferred statistics.
  • With 5 k-sensors and 9 u-sensors, PI-GANs achieved comparable accuracy to the method proposed in [32].

5.4. Multiple groups of training data

The multiple-group experiment tests whether PI-GANs can combine independently collected snapshots that cannot be aligned across sensor groups. Using multiple discriminators, the method reduced error relative to training on the first group alone.

  • Two groups provided training data: 13 k-sensors, 2 boundary u-sensors, and 13 f-sensors in group 1, plus one u-sensor at x = 0 in group 2.Each group contained 1000 snapshots.
  • A single snapshot from group 2 is almost useless because it cannot be aligned with a snapshot from group 1.The ensemble of group-2 snapshots can still describe the distribution of u(x; ω) at x = 0.
  • The error decreased compared with using only the first data group, demonstrating use of ensemble information from independently collected snapshots.The authors state that this case was, to their knowledge, uniquely manageable by their method at the time.
  • Using two discriminators, PI-GANs incorporated both groups and estimated the mean and standard deviation of k(x; ω) and u(x; ω).The experiment used input-noise dimension 20 and selected 33 generators for evaluation.

6. Summary and future work

The paper presents PI-GANs as a unified approach for stochastic differential equations using physics-informed generators and WGAN-GP-based training. Experiments cover Gaussian-process learning, forward, inverse, mixed, high-dimensional, and multiple-data-group settings, while identifying computational and uncertainty-quantification limitations.

  • PI-GANs combine feed-forward DNNs with an SDE-induced neural network to approximate stochastic processes from limited scattered measurements.The framework uses WGAN-GP losses and supports multiple discriminators for independently collected data groups.
  • WGAN-GP approximated Gaussian processes despite mismatched input-noise and target effective dimensionalities, whereas vanilla GANs failed for deterministic-boundary processes.Approximation improved with more sensors and snapshots, and overfitting occurred in both generators and discriminators.
  • Without changing the framework, PI-GANs solved forward, inverse, and mixed elliptic SDE problems while matching benchmark means and standard deviations.The approximated processes included the solution, forcing, and diffusion coefficient.
  • 30 dimensions were demonstrated for the stochastic forcing, with inferred solution statistics and forcing spectra matching reference values well.The paper also reports improved error when a second, independently collected sensor group was added.
  • Limited training data produced overfitting in both discriminators and generators, while PI-GANs also have higher training cost than physics-informed neural networks.The paper does not yet quantify measurement uncertainty or GAN approximability uncertainty.
  • The authors estimate low-polynomial computational growth with problem dimensionality but leave systematic very-high-dimensional scalability studies for future work.
Loading 1811.02033v1…