PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-Informed Diffusion Models

Jan-Hendrik Bastek, WaiChing Sun, Dennis M. Kochmann

arXiv:2403.14404v4cs.LGcs.CE

TL;DR

Scientific diffusion models must represent complex data distributions while ensuring generated samples satisfy known governing equations. This paper adds a first-principles residual loss to diffusion training; across fluid-flow and topology-optimization case studies, it reports improved constraint-related outcomes and evidence of reduced overfitting.

  • Problem

    Diffusion models trained only on data do not strictly enforce known governing equations, although scientific deployment requires generated samples to meet the underlying physics.

    Source: §1 · pp. 1–2

  • Method

    The framework adds a first-principles residual loss to the data-driven diffusion objective, incorporating PDE constraints during training.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    PIDM outperforms data-driven and prior frameworks across Darcy flow and topology optimization; in Darcy flow, residual error falls around two orders of magnitude versus the standard diffusion model.

    Source: §4.1 · p. 8, §6 · p. 10, §4.2 · p. 10

  • Takeaways & Limitations

    The training objective can be added to existing diffusion architectures without changing inference and can address equality and inequality constraints and auxiliary optimization objectives.

    Source: §1 · p. 2

    Residual information may remain beneficial even with infinite training samples because it typically imposes constraints on higher-order derivatives.

    Source: §3 · p. 3

Abstract

from arXiv · show

Generative models such as denoising diffusion models are quickly advancing their ability to approximate highly complex data distributions. They are also increasingly leveraged in scientific machine learning, where samples from the implied data distribution are expected to adhere to specific governing equations. We present a framework that unifies generative modeling and partial differential equation fulfillment by introducing a first-principle-based loss term that enforces generated samples to fulfill the underlying physical constraints. Our approach reduces the residual error by up to two orders of magnitude compared to previous work in a fluid flow case study and outperforms task-specific frameworks in relevant metrics for structural topology optimization. We also present numerical evidence that our extended training objective acts as a natural regularization mechanism against overfitting. Our framework is simple to implement and versatile in its applicability for imposing equality and inequality constraints as well as auxiliary optimization objectives.

1 INTRODUCTION

Diffusion models can learn complex scientific data distributions, but a data-driven objective does not strictly enforce known governing equations, so generated samples may lack the precision needed for deployment. The paper introduces physics-informed diffusion models that incorporate PDE constraints during training and reports improved residuals and evidence of reduced overfitting.

1 INTRODUCTION

  • Diffusion models can represent distributions of feasible solutions and designs, rather than returning only one deterministic outcome, in scientific inverse problems.Examples include low-fidelity data upscaling, molecule and material design, and metamaterial design for target stress-strain responses.Source: §1 · p. 1
  • Known governing equations are often explicit in scientific applications, yet traditional diffusion training remains data-driven and does not strictly enforce those constraints.Training data may come from simulations that fulfill the equations, but generated samples can still fall short of required precision for deployment.Source: §1 · pp. 1–2
  • The framework unifies diffusion models with PINNs by incorporating PDE constraints into training, reducing generated-sample residuals and showing evidence of less overfitting.The authors also report that inference is unaffected and that the framework can address equality and inequality constraints and auxiliary optimization objectives.Source: §1 · p. 2

2 BACKGROUND

The background describes diffusion models as learned reverse processes for generating data and frames physical constraints as PDEs and boundary conditions on solution fields. It defines residuals as measures of how generated fields satisfy those equations.

2.1 DENOISING DIFFUSION MODELS

  • Diffusion models generate data by learning a neural reverse process that approximates the inverse of a Gaussian-noising process, with training based on a variational lower bound.The model can predict the clean signal, noise, or mean; the clean-signal objective uses a time-weighted mismatch.Source: §2.1 · pp. 2–3

2.2 ASSEMBLY OF GOVERNING EQUATIONS

  • The physical problem is expressed through PDEs in the domain and boundary conditions on its boundary, with the solution field required to satisfy both.The differential operator F acts on the field in the domain, while B specifies boundary constraints.Source: §2.2 · p. 3
  • For image-based models, the solution field is represented on a discretized pixel grid, and finite differences or comparable methods approximate derivatives used to evaluate its residual.The residual stacks PDE and boundary-condition discrepancies; evaluation uses the mean absolute error RMAE.Source: §2.2 · p. 3

3 PHYSICS-INFORMED DIFFUSION MODELS

The method formulates physical constraints as residual likelihoods and combines them with the data likelihood so generated distributions remain informed by observed data. It simplifies training by evaluating residuals on clean-signal estimates and varying constraint strength with denoising time.

3.1 CONSIDERATION OF PDE CONSTRAINTS

  • The method treats zero residuals as virtual observations and maximizes their likelihood, interpreting PDE fulfillment probabilistically rather than as a separate post-processing step.In the zero-variance limit, the virtual likelihood recovers strict residual enforcement.Source: §3.1 · p. 4

3.2 CONSIDERATION OF OBSERVED DATA

  • The objective includes both the usual data likelihood and the virtual residual likelihood, because data guide the model toward a relevant solution subset and help avoid collapse to one feasible solution.The paper states that the combined score-based objective is consistent under its stated interpretation, with the optimal score model also maximizing the virtual likelihood and recovering samples from the data distribution.Source: §3.2 · p. 4

3.3 SIMPLIFICATION OF THE TRAINING OBJECTIVE

  • Residuals are evaluated on either the conditional-mean clean-signal estimate or an actual sample obtained by accelerated DDIM, avoiding costly full sampling during training.Mean estimates can create a Jensen gap because the residual of the expectation need not equal the expected residual; sample estimation costs more computation and trades sample quality against DDIM timesteps.Source: §3.3 · pp. 4–5
  • The residual-likelihood variance is increased for noisier timesteps and tightened as t approaches zero, reflecting lower confidence in estimates from noisier inputs.Figure 1 describes clean-signal approximations at any denoising timestep using either the conditional expectation or accelerated DDIM sampling.Source: §3.3 · p. 5, Figure 1 · p. 5
  • The resulting PIDM training objective extends standard diffusion training with a physics-informed residual loss and requires only minor modifications to the existing setup.The objective places the data and residual contributions under the same expectation.Source: §3.3 · p. 6
  • The residual term can encode differentiable equality or inequality constraints and auxiliary objectives, while its scaling c must balance constraint enforcement against the data likelihood.The paper notes that no notable improvement was found from multiple intermediate timesteps, which require additional forward passes.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments test physics-informed diffusion on Darcy flow and structural topology optimization. PIDM improves physical residuals across both settings while retaining data-distribution performance and avoiding auxiliary datasets or surrogate models in topology optimization.

4 EXPERIMENTS

  • The experiments cover two benchmarks in which generated samples are expected to satisfy governing PDEs and constraints.A toy problem is also provided for intuition about the proposed loss.Source: §4 · p. 6

4.1 DARCY FLOW

  • Darcy flow uses simulated permeability-pressure pairs on a 64 × 64 grid, with 10,000 training and 1,000 validation examples.The dataset is generated by solving the governing equations for sampled permeability fields.Source: §4.1 · p. 6

4 EXPERIMENTS

  • About two orders of magnitude lower Darcy residuals and 33%/35% topology residual reductions versus standard diffusion anchor PIDM’s cross-case gains.The topology reductions are on the in- and out-of-distribution test sets, respectively.Source: Figure 2 · p. 7, §4.1 · p. 8, Table 1 · p. 9

4.1 DARCY FLOW

  • PG-Diffusion showed no significant residual reduction, while CoCoGen improved moderately but remained far from physical consistency.PIDM’s mean estimation outperformed sample estimation on residual error; both retained generative diversity and data-distribution alignment.Source: Figure 2 · p. 7, §4.1 · p. 8

4.2 TOPOLOGY OPTIMIZATION

  • Topology optimization benchmarks use 30,000 optimized structures and in- and out-of-distribution boundary conditions; PIDM adds no auxiliary data or models.PIDM training takes longer, but inference remains identical to standard diffusion and requires no surrogate model.Source: §4.2 · pp. 8–9
  • PIDM significantly outperforms prior topology frameworks in compliance minimization, with only a slight increase in volume fraction error; standard diffusion’s higher volume error limits comparability.The comparison uses unprocessed samples from Giannone et al. (2023).Source: Table 1 · p. 9, §4.2 · p. 10

5 RELATED WORK

The related work compares PIDM with physical-constraint methods and places it alongside approaches that alter conditioning or inference. It contrasts training-time constraint learning with post-processing and auxiliary-model strategies.

5 RELATED WORK

6 CONCLUSION

The conclusion presents PIDM as a way to incorporate physical constraints into diffusion training. The studies show improved performance across two case studies, while the authors identify grid-related scope limits and directions for future work.

6 CONCLUSION

  • PIDM unifies diffusion modeling with a physics-informed paradigm so models internalize constraints that generated samples must satisfy.The conclusion reports favorable results in two case studies and numerical evidence of less overfitting.Source: §6 · p. 10
  • The studies are restricted to geometrically simple domains because they require architectures with consistent residual evaluation on fixed regular grids.The authors suggest graph-based architectures for arbitrary meshes or implicit coordinate encodings as possible remedies.Source: §6 · p. 10
  • Future work may explore more sophisticated virtual-likelihood variance schedulers and coordinate-based representations, which enable exact gradient calculation at increased computational cost.Source: §6 · p. 10

A.1 CONSISTENCY

The consistency argument shows that, under ideal assumptions, adding a virtual residual likelihood preserves recovery of the true distribution. It also distinguishes this ideal result from numerical discretization and intermediate-latent residual constraints.

A.1 CONSISTENCY

  • The score-matching objective extended with a virtual residual likelihood remains consistent in the ideal setting.The argument assumes constrained data samples and an optimal score; the true distribution minimizes the virtual likelihood objective.Source: §A.1 · p. 16
  • With the optimal score, reverse-SDE sampling from p(xT) recovers p(x0) when the data samples satisfy R(x0) = 0.The proposition assumes the reverse SDE is solved as stated and p(xT) matches the corresponding terminal distribution.Source: §A.1 · p. 16
  • DDPM and DDIM are both available discretizations of the reverse process, and the optimal score is unaffected by that choice.The reverse diffusion process is not unique.Source: §A.1 · p. 16
  • Discretizing the reverse SDE introduces bias because equation 16 has no closed-form solution.This numerical approximation can introduce bias even with a perfect score model.Source: §A.1 · p. 16
  • Enforcing R(xt) = 0 on intermediate latent variables can conflict with the underlying marginal distribution.The stated inconsistency arises because the intermediate latent residual need not be zero.Source: §A.1 · p. 17

A.2 MEAN ESTIMATION FOR SCORE-BASED MODELS

For score-based models, the predicted clean sample is the conditional mean given a noisy input, and evaluating its residual can create an inconsistent objective.

A.2 MEAN ESTIMATION FOR SCORE-BASED MODELS

  • The estimate ˆx0 represents E[x0|xt] for the specified Gaussian forward conditional marginal.The text attributes this conditional-mean expression to Kingma et al. (2021).Source: §A.2 · p. 17
  • The passage states that x̂0 is obtained as E[x0|xt], the conditional mean of x0 given xt, and cites Kingma et al. (2021) as an example.Source: §A.2 · p. 17
  • The estimate x̂0 is the conditional expectation E[x0|xt], as noted with reference to Kingma et al. (2021).Source: §A.2 · p. 17

A.3 DETAILS ON THE SIMPLIFIED TRAINING OBJECTIVE

The training objective simplifies latent sampling by using q(x1:T) instead of the learned distribution pθ(x1:T). The simplification is justified by an assumption about variational-bound optimization.

A.3 DETAILS ON THE SIMPLIFIED TRAINING OBJECTIVE

  • The objective requiring samples from pθ(x1:T) is simplified by sampling instead from the available q(x1:T).This amounts to ignoring the likelihood ratio.Source: §A.3 · p. 17
  • The simplification assumes that optimizing the variational bound brings pθ(x1:T) close to q(x1:T), reducing the bias as training proceeds.Source: §A.3 · p. 17
  • Evaluating the simplified expression yields the physics-informed diffusion model loss in equation 14.Source: §A.3 · p. 17

A.4 DDIM SAMPLING

The DDIM sampling procedure uses a reduced subsequence of denoising steps to estimate x0 during training. The selected two-step procedure limits forward passes while accounting for noisier estimates.

A.4 DDIM SAMPLING

  • The chosen sequence τ = [1, t] samples x1 from xt and then obtains x0, using two model forward passes.Source: §A.4 · p. 17
  • Providing multiple intermediate timesteps showed no notable empirical improvement and required additional forward passes.Source: §A.4 · p. 17
  • The virtual residual likelihood penalizes deviations less at noisier timesteps because its variance is increased.Source: §A.4 · p. 17

A.5 OTHER CONSTRAINT TYPES AND AUXILIARY OPTIMIZATION OBJECTIVES

The appendix extends the constraint and objective formulations, then details toy, Darcy-flow, and topology-optimization studies. The toy study illustrates the residual–diversity trade-off, while later sections describe implementation and evaluation choices.

A.5.1 INEQUALITY CONSTRAINTS

  • Inequality constraints use a ReLU penalty for violations, incorporated into the variational loss through a Gaussian virtual likelihood with target zero.The penalty is max(0, h(x0) − hmax).Source: §A.5.1 · p. 18

A.5.2 OPTIMIZATION OBJECTIVES

  • For auxiliary optimization objectives, an unknown optimum is handled through a zero-target pseudo-observable and an exponential distribution; the loss weighting trades off objectives.The exponential-distribution parameter λ is kept constant here.Source: §A.5.2 · p. 18

A.6.1 TOY PROBLEM

  • PIDM lowers toy-circle residual error versus standard diffusion after about 100–120 epochs and aligns samples better; residual-only training collapses to one feasible point.The pattern is similar with mean and sample estimation; an uninformative prior also approximates the target distribution, but with reduced accuracy.Source: Figure 5 · p. 19, §A.6.1 · pp. 19–20, Figure 6 · p. 20
  • Increasing c lowers residual error but concentrates samples; c = 0.005 balances diversity and residual, while c ≫ 1 ignores data loss and collapses.The c sweep uses sample estimation and compares six scale factors.Source: Figure 7 · p. 20, §A.6.1 · p. 21

A.6.2 DETAILS OF THE DARCY FLOW STUDY

  • The Darcy study samples permeability from a Gaussian random field, truncates its Karhunen–Loève expansion to 64 terms, and solves pressure on a 64 × 64 grid.The overdetermined pressure system imposes boundary conditions and an integral constraint.Source: §A.6.2 · p. 21

A.6.3 DETAILS OF THE TOPOLOGY OPTIMIZATION STUDY

  • The topology study conditions generation on mechanics fields and boundary data, predicts displacements and density, and evaluates residuals with the finite-element stiffness matrix.Finite differences would introduce inconsistencies with the FEM training data; nodal quantities are interpolated to a 65 × 65 grid for residual computation.Source: §A.6.3 · p. 23
  • For CoCoGen comparison, the full residual Jacobian exceeded GPU memory, so the study used constant scaling and selected hyperparameters by median in-distribution residual.The appendix reports training times of approximately 13–22 hours for Darcy models and 48–54 hours for topology models.Source: §A.6.3 · p. 23, §A.7 · p. 24

A.8.1 DARCY FLOW; A.8.2 TOPOLOGY OPTIMIZATION

  • Additional figures show generated Darcy permeability and pressure fields and topology designs, including residuals and compliance or volume measures.The topology examples are conditioned on the out-of-distribution test set.Source: Figure 8 · p. 25, Figure 9 · p. 26
Loading 2403.14404v4…