Enforcing Analytic Constraints in Neural-Networks Emulating Physical Systems

Tom Beucler; Michael Pritchard; Stephan Rasp; Jordan Ott; Pierre Baldi; Pierre Gentine

Source-linked AI summary

Enforcing Analytic Constraints in Neural-Networks Emulating Physical Systems

Tom Beucler, Michael Pritchard, Stephan Rasp, Jordan Ott, Pierre Baldi, Pierre Gentine

arXiv:1909.00912v5Sep 3, 2019physics.comp-phphysics.ao-ph

TL;DR

Neural networks can accurately emulate physical systems while violating analytic constraints that should hold exactly. This paper introduces architecture- and loss-based methods for enforcing such constraints, and in climate modeling finds that architectural enforcement preserves performance while improving constrained outputs.

Problem
Neural-network emulators may violate exact physical constraints, threatening physically consistent modeling of systems such as climate convection.
Method
The paper introduces ACnets, which enforce analytic constraints through fixed architectural layers, and compares them with unconstrained and loss-constrained networks.
Results
ACnets perform within 3% of the lowest-MSE UCnet on average while satisfying constraints to approximately machine precision.
Takeaways & Limitations
Enforcing constraints systematically reduces errors in variables appearing in the constraint equations, although not necessarily for hard-to-predict variables such as precipitation.

Abstract

from arXiv · show

Neural networks can emulate nonlinear physical systems with high accuracy, yet they may produce physically-inconsistent results when violating fundamental constraints. Here, we introduce a systematic way of enforcing nonlinear analytic constraints in neural networks via constraints in the architecture or the loss function. Applied to convective processes for climate modeling, architectural constraints enforce conservation laws to within machine precision without degrading performance. Enforcing constraints also reduces errors in the subsets of the outputs most impacted by the constraints.

I. INTRODUCTION

The paper addresses physically inconsistent neural-network predictions by introducing a systematic framework for analytic constraints, including a formulation that converts nonlinear constraints into linearly constrained mappings. The approach is designed for constraints that may involve inputs, outputs, and additional parameters.

Motivation: Data-driven models can violate exact analytic constraints, including conservation laws, even while optimizing overall predictive performance.Such violations may occur on individual samples despite high aggregate accuracy.
Contribution: ACnet systematically enforces analytic constraints in more general physical systems than prior hard-constraint approaches limited to specific governing equations.The paper compares architecture-constrained, unconstrained, and loss-constrained neural networks in climate modeling.
Framework: A constraints matrix C represents the resulting linear system, whose independent constraints must be fewer than the combined input and output dimensions for non-unique regression solutions.The matrix has rank n and belongs to R^n × R^(m+p).
Framework: The framework defines temporary neural-network inputs and outputs, then rewrites constraints as an identically zero function before selecting linearly constrained variables.Non-dimensionalization is recommended to support constraint design, interpretation, and loss-function performance.
Framework: The framework supports nonlinear and differential constraints, including differential constraints discretized with a forward Euler timestep Δt.The choices of transformed inputs, outputs, and constraints matrix are not unique.

B. Enforcing the Constraints

The paper compares soft loss penalties with hard architectural enforcement of constraints. ACnet preserves the standard MSE objective while computing selected outputs as exact residuals from fixed constraint layers.

Baseline: The unconstrained baseline optimizes a multilayer perceptron with mean-squared error as its loss function.The network is trained on preexisting measurements of inputs and outputs.
Loss-constrained networks: LCnet imposes soft constraints by adding the mean-squared constraint residual P to the MSE with weight α.Its loss is L(α) = αP(x, yNN) + (1−α)MSE(yTruth, yNN).
Architecture-constrained networks: ACnet imposes hard constraints by augmenting an optimizable neural network with n fixed conservation layers while retaining MSE as the loss.The fixed layers sequentially enforce the constraints to within machine precision.
Architecture-constrained networks: ACnet predicts p−n direct output components and computes the remaining n components as exact residuals from the constraints.The full output and its gradients pass through the constraint layers during optimization, so the learned parameters depend on the constraints.

C. Linking Constraints to Performance

The paper links constraint violation to prediction error through an error-vector decomposition under the assumption that the true outputs exactly satisfy the constraints. For ACnets, the constraint relation forces a negative cross-term rather than guaranteeing lower overall error.

Analytic link: The analysis decomposes neural-network predictions into truth and error vectors and assumes the true system satisfies the constraints exactly.This assumption treats measurement errors as absent in the constraint relation.
Analytic link: Equation 8 relates the amount of constraint violation to the prediction error vector.The relation connects physical inconsistency with errors in outputs appearing in the constraint equations.
Analytic link: For each constraint, squaring the relation links violation to componentwise squared error and a cross-term.The squared-error term is positive-definite.
Architecture-constrained implication: In ACnets, exact constraint enforcement sets the left-hand side to zero within numerical errors, making the cross-term negative.The cross-term cannot generally be predicted before optimization, so the equation does not provide an a-priori performance prediction.

III. APPLICATION

The application targets neural-network parameterization of subgrid convection and clouds, where conservation errors matter for long-term climate simulation. ACnet uses fixed constraint layers to enforce four column-integrated conservation laws while predicting high-dimensional convective tendencies and fluxes.

Climate application: Subgrid parameterization is a major source of error and uncertainty in numerical weather and climate prediction.The application uses machine learning to emulate fine-scale convective effects on coarse-scale atmospheric dynamics.
Architecture: ACnet computes direct outputs with a standard neural network and obtains remaining outputs as residuals through fixed constraint layers.This architecture is the mechanism used to enforce the application’s conservation laws.
Conservation targets: The model constrains column-integrated energy, mass, longwave radiation, and shortwave radiation.These correspond to enthalpy, water, longwave-radiation, and shortwave-radiation conservation laws.
Model formulation: The climate parameterization maps a 304-dimensional local-state input vector to a 216-dimensional output vector containing convective tendencies and related variables.The inputs include vertical profiles, prescribed large-scale conditions, and scalars; outputs include tendencies, radiative terms, and precipitation.
Constraint construction: The conservation laws are written as a sparse 4 × (304 + 218) matrix acting on the inputs and outputs.Variables are normalized and nondimensionalized using 1 W m−2 before constructing the constraints.

D. Implementation

The study compares unconstrained, loss-constrained, and architecture-constrained networks using prediction error and constraint violation, finding that hard architectural constraints preserve performance while improving constrained outputs.

Performance vs Conservation: Increasing the loss weight α produces a monotonic trade-off between prediction performance and constraint satisfaction.MSE and constraint-violation measure P show this trade-off across training, validation, and test sets.
Performance vs Conservation: ACnet performs within 3% of the lowest-MSE UCnet while satisfying physical constraints to approximately machine precision.The result holds across training, validation, and test sets.
Constraint-Related Errors: ACnet predicts the thermodynamic term more accurately than all other neural networks, with accuracy closely related to enthalpy-constraint violation.LCnet follows ACnet in the plotted comparison.
Constraint-Related Errors: Constraint enforcement reduces errors in variables appearing in the constraints, but not necessarily in individual components or difficult-to-predict outputs such as precipitation.Conservation does not guarantee generalization far outside the training distribution, including in the Tropics of a warmer climate.
Nonlinear Constraints: ACnets can also enforce nonlinear constraints through the proposed conversion-layer framework.

Supplemental Material Enforcing Analytic Constraints in Neural-Networks Emulating Physical Systems

The supplemental material is organized into five independently readable sections covering application variables, ACnet implementation, architecture comparisons, nonlinear constraints, and inequality constraints.

Section A introduces the variables and constraints specific to the climate-modeling application.
Section B details implementation of the architecture-constrained network and its residual-output choices.
Section C compares different network types and architectures.
Section D presents an example of enforcing nonlinear constraints in neural networks emulating physical systems.
Section E discusses how to enforce inequality constraints.

A. Derivation of Dimensionless Conservation Equations

The climate application formulates energy, mass, and radiative conservation laws for atmospheric columns, then nondimensionalizes and discretizes them into linear constraints on network variables.

Conservation Laws: The atmospheric model conserves liquid/ice water static energy and total water, recast as energy and mass conservation.
Conservation Laws: Longwave and shortwave radiation are conserved by balancing boundary radiative flux differences with vertically integrated atmospheric heating.
Conservation of Energy: Column enthalpy is defined as the mass-weighted vertical integral of sensible and latent heat, with ice as the zero-energy reference.
Conservation of Energy: The enthalpy formulation uses atmospheric pressure, gravity, temperature, humidity, liquid water, and thermodynamic constants to define specific enthalpy.
Conservation of Water: The water-conservation equation balances changes in total column water against surface evaporation and precipitation.
Nondimensionalization and Discretization: Vertical profiles are discretized into 30 pressure-weighted levels, converting continuous conservation equations into linear constraints on discrete variables.All tendencies are converted to W m−2 and divided by 1 W m−2 for nondimensionalization.

B. Implementation of the Architecture-constrained Network

The standard ACnet implementation selects direct outputs from an optimizable network and computes residual outputs from a fixed constraints matrix, using conservation layers to enforce each law.

Standard Implementation: The implementation examines application-specific ACnets, residual-output sensitivity, and preferential loss weighting to reduce residual-output bias.
Output Construction: ACnet chooses p−n direct outputs and n residual outputs computed from the fixed constraints matrix C.When q outputs appear in the constraints, there are n!/[q!(q−n)!] possible residual-output choices.
Output Construction: Residual outputs can be computed either by custom conservation layers or inside a modified loss function; the standard implementation uses custom layers.
Output Construction: The standard network first outputs 214 direct components, then solves for the remaining residual components using C.
Conservation Layers: One conservation layer can solve each row of C, although all constraints could alternatively be grouped into a single layer.
Conservation Layers: The four conservation layers compute residuals for shortwave flux, longwave flux, lowest-level humidity tendency, and lowest-level temperature tendency.

B.2. Sensitivity to Residual Index

ACnets conserve column mass, energy, and radiation across residual-index choices, but the selected residual levels develop larger squared errors. Weighting residual outputs can reduce this localized bias, with a trade-off against total MSE.

Residual-index sensitivity: Residual-index choices are ACnet hyperparameters, including the vertical level and variable whose output is computed as a constraint residual.The reference configuration uses the lowest-level convective heating and moistening tendencies, but other levels or variables can be selected.
Residual-index sensitivity: 10−9W2m−4 conservation penalty: all ACnets conserve column mass, energy, and radiation across training, validation, and test sets.The tested ACnets vary the residual levels used for mass and enthalpy conservation.
Residual-index sensitivity: Each ACnet shows larger squared error at its selected residual level despite similar overall MSEs, with the effect appearing in both validation and test sets.For example, q14T14 has its largest relative error at the middle residual level, while q29T29 and q0T29 show larger heating error at the lowest level.
Residual-index sensitivity: Residual-level squared error generally follows variance by vertical level, except near the surface and in the upper atmosphere.Near the surface, part of the signal is not deterministically predictable; in the upper atmosphere, convective-moistening variance is very small.
Residual-bias weighting: β = 2: a moderate residual-output loss multiplier can remove the residual-level anomaly while increasing total MSE.Increasing β reduces residual bias but gives direct outputs less weight, raising overall error; β can be tuned with residual indices.

C.2 Formal Hyperparameter Optimization

The appendix tunes network hyperparameters, compares architectural constraints with post-processing, and extends the approach to nonlinear constraints using humidity conversion layers.

C.2 Formal Hyperparameter Optimization: Four hyperparameter searches use SHERPA random searches with more than 200 trial models each, selecting the lowest validation-set MSE.Each trial trains for at least 20 epochs.
C.2 Formal Hyperparameter Optimization: The best UCnet baseline uses 5 layers of 512 nodes, Leaky ReLU coefficient 0.3, no dropout, and no batch normalization.Training and validation MSE plateau after 20 epochs, motivating that stopping point.
C.2 Formal Hyperparameter Optimization: LCnet searches find that increasing α typically raises MSE, while α ≈0.01 gives the lowest-MSE LCnets but still exceeds the best UCnet MSE.This reflects a trade-off between performance and physical constraints.
C.2 Formal Hyperparameter Optimization: ACnet tuning favors z1 > 20 and β > 2, while residual performance depends jointly on β, z1, and z2.The five lowest-validation-MSE networks are evaluated on test squared error and Log.Bias.
C.3. Advantages over enforcing constraints after training: Post-processing computes residual outputs after training, whereas ACnet constraint layers pass full-output loss gradients through the architecture during optimization.Direct-output MSEs are systematically within 3% between the approaches.
C.3. Advantages over enforcing constraints after training: ACnets expose residual outputs to data during training, producing lower residual-output MSEs than post-processed UCnets in nearly all comparisons.Overall ACnet MSE is always smaller but within 3% of post-processed UCnet MSE; one q4T4β1.35 case is an exception for ˙T4.
D. Extending to Nonlinear Constraints: An Example: The nonlinear extension maps relative-humidity inputs and outputs through specific-humidity conversion layers before and after the constrained network.ACnetNL uses the same conversion strategy as UCnetNL while replacing UCnet with ACnet.
D.1 Formulation: Relative humidity is analytically defined from water-vapor partial pressure and saturation pressure, enabling the conversion-layer formulation.The appendix gives the definition and constants Rv ≈461 J kg−1 K−1 and Rd ≈287 J kg−1 K−1.

D.2 Results

The nonlinear comparison evaluates unconstrained and constrained networks with humidity conversion layers on climate-modeling mappings. ACnetNL achieves substantially smaller constraint penalties while retaining similar predictive performance.

D.2 Results: Three network types—UCnet, UCnetNL, and ACnetNL—are compared for performance and constraint penalties.UCnetNL isolates the optimization effect of conversion layers, while ACnetNL applies nonlinear architectural constraints.
D.2 Results: Nine networks are trained for 20 epochs with RMSprop, retaining the state with minimal validation loss to limit overfitting.Each network type contributes three trained networks.
D.2 Results: ACnetNL’s constraint penalty is 8 orders of magnitude smaller than UCnetNL’s on both validation and test sets.Performance and penalties are reported in Table IX and complemented by vertical squared-error profiles.
D.2 Results: UCnetNL’s constraint penalty is more than one order of magnitude lower than UCnet’s, indicating that conversion layers affect unconstrained optimization.The passage interprets this as easier approximation of linear than strongly nonlinear constraints by unconstrained networks.
D.2 Results: Networks using nonlinear conversion layers have MSEs higher by a factor ∼1.5 than UCnets, indicating harder optimization.The result is described as a training trade-off associated with the conversion layers.
D.2 Results: The absolute MSE difference between UCnetNL and ACnetNL is smaller than 10 W2m−4 while ACnetNL provides machine-precision nonlinear constraint adherence.The reported trade-off is framed as a small performance cost for stronger constraint satisfaction.

D.3 Saturation pressure of water vapor

The appendix computes saturation vapor pressure analytically to convert between specific and relative humidity. Its temperature-dependent formulation switches between liquid and ice regimes and blends them across an intermediate range.

D.3 Saturation pressure of water vapor: Saturation vapor pressure is obtained by integrating the Clausius–Clapeyron equation under the model’s microphysical assumptions.The resulting analytic expression supports conversion between specific and relative humidities.
D.3 Saturation pressure of water vapor: For T > T0, saturation pressure uses the liquid-water expression eliq(T), while for T < T00 it uses the ice expression eice(T).The temperature thresholds are T0 = 273.16 K and T00 = 253.16 K.
D.3 Saturation pressure of water vapor: The liquid and ice saturation pressures use different polynomial approximations with temperature switches and coefficient vectors.The liquid approximation uses aliq, while the ice approximation uses aice and cice coefficients.
D.3 Saturation pressure of water vapor: Figure 8 compares squared errors in convective moistening ˙qv and heating ˙T versus pressure across validation and test sets.Full lines show medians, shaded regions show first-to-third quartiles, and colors distinguish NN types.
D.3 Saturation pressure of water vapor: The reported relative-humidity-tendency MSE values are smaller because Table IX calculates MSE using relative humidity tendencies.The table organizes validation metrics across UCnet, UCnetNL, and ACnetNL.
D.3 Saturation pressure of water vapor: The implementation includes a ramp function and numerical documentation for the saturation-pressure calculation.The cited implementation documents the relative-humidity conversion details.
D.3 Saturation pressure of water vapor: Between T00 and T0, saturation pressure is a weighted mean of eliq and eice with weight ω linear in absolute temperature.This provides the intermediate-temperature transition between liquid and ice saturation pressures.
D.3 Saturation pressure of water vapor: The inequality-constraint example enforces positive liquid-water concentration before the equality-constraint layer, preserving equality constraints afterward.The inequality layer transforms the tendency so the current-timestep concentration remains positive.