PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles

Xiaowei Jia, Jared Willard, Anuj Karpatne, Jordan S Read, Jacob A Zwart, Michael Steinbach, Vipin Kumar

arXiv:2001.11086v3cs.LGeess.SP

TL;DR

Physics-based models are limited by approximate process representations and difficult parameter calibration, whereas black-box ML models can require much data and violate physical laws. The paper proposes PGRNN, which combines recurrent neural networks with physics-based models and conservation laws. PGRNN achieves high prediction performance with fewer observations used for refinement than a calibrated parameterized physics-based model.

  • Problem

    Physics-based models are approximate and difficult to calibrate, while black-box ML models can require abundant data and produce physically inconsistent results.

  • Method

    PGRNN combines recurrent neural networks with physics-based models, using recurrent memory and explicit physical laws such as energy conservation.

  • Results

    PGRNN obtains high prediction performance with fewer observation data used for refinement than a parameterized physics-based model calibrated using many observations.

  • Takeaways & Limitations

    PGRNN provides a framework for modeling spatial and temporal physical processes while incorporating energy conservation.

  • Takeaways & Limitations

    Future research must determine which types of dynamical systems models are suitable for this approach.

Abstract

from arXiv · show

Physics-based models of dynamical systems are often used to study engineering and environmental systems. Despite their extensive use, these models have several well-known limitations due to simplified representations of the physical processes being modeled or challenges in selecting appropriate parameters. While-state-of-the-art machine learning models can sometimes outperform physics-based models given ample amount of training data, they can produce results that are physically inconsistent. This paper proposes a physics-guided recurrent neural network model (PGRNN) that combines RNNs and physics-based models to leverage their complementary strengths and improves the modeling of physical processes. Specifically, we show that a PGRNN can improve prediction accuracy over that of physics-based models, while generating outputs consistent with physical laws. An important aspect of our PGRNN approach lies in its ability to incorporate the knowledge encoded in physics-based models. This allows training the PGRNN model using very few true observed data while also ensuring high prediction accuracy. Although we present and evaluate this methodology in the context of modeling the dynamics of temperature in lakes, it is applicable more widely to a range of scientific and engineering disciplines where physics-based (also known as mechanistic) models are used, e.g., climate science, materials science, computational chemistry, and biomedicine.

1 INTRODUCTION

Physics-based models are valuable but approximate, parameter-intensive, and difficult to calibrate, while black-box ML models require abundant data and may violate physical laws. The paper introduces PGRNN, combining recurrent learning with physics-based knowledge to model physical processes accurately with sparse observations.

  • Challenges with existing models: Physics-based models approximate reality because of simplified process representations and uncertain parameter selection.Their parameters often require calibration from limited observations, and exhaustive calibration can be computationally costly and prone to over-fitting.
  • Challenges with existing models: Black-box ML models can capture complex patterns but require substantial labeled data and may produce physically inconsistent outputs.They also may fail to generalize to scenarios outside the training distribution, such as future warmer climates.
  • Prior approaches: Existing physics-guided approaches may impose output constraints but cannot generally represent internal physical states or overcome training-data scarcity.The paper identifies energy conservation and limited observations as examples of these unresolved challenges.
  • Proposed framework: PGRNN combines recurrent neural networks with explicit physical laws, including energy and mass conservation, by tracking physical states in its recurrent structure.The framework uses architectures such as LSTM to preserve long-term temporal dependencies.
  • Proposed framework: Physics-based penalties support robust training when observed outputs are available at only a small number of time steps.The approach uses physics-based knowledge to complement data-driven learning and improve physical consistency.
  • Results and scope: Pre-training on simulated data enables PGRNN to outperform a state-of-the-art physics-based model using relatively few observed data for refinement.The benefit persists when simulations come from lakes differing from the target in geometry, clarity, or climate.

2 PROBLEM FORMULATION AND PRELIMINARIES

The paper formulates lake-temperature prediction across depths and time from meteorological drivers and sparse observations, then reviews GLM and sequential machine-learning approaches. GLM represents lake thermodynamics but requires calibration and retains simplifications, while a global depth-aware model can use observations jointly across depths.

  • 2.1 Problem formulation: The task predicts water temperature at each depth and date from physical input variables and sparse sensor observations.Inputs include meteorological conditions, depth, and day of year; observed temperatures are available only for certain dates and depths.
  • 2.1 Problem formulation: The problem is treated as one-dimensional temperature modeling because depth is the single modeled spatial dimension.
  • 2.1 Problem formulation: The input features include short-wave and long-wave radiation, wind speed, air temperature, relative humidity, rain, snow, depth, and day of year.
  • 2.2 General Lake Model (GLM): GLM simulates lake heating, radiation attenuation, vertical mixing, evaporation, and outgoing long-wave radiation.These processes govern lake-water temperature dynamics and are represented through parameters such as vertical mixing, wind sheltering, and water clarity.
  • 2.2 General Lake Model (GLM): GLM calibration searches parameter combinations to minimize model error, but can be laborious and computationally intensive.Even with ample training data, calibration remains limited by simplified representations and rigid formulations.
  • 2.3 Machine learning model for sequential data: Black-box sequential models learn transformations from input series to target series, but sparse observations make separate models for each depth infeasible.
  • 2.3 Machine learning model for sequential data: A global model uses depth as an input feature, allowing observations from any depth and time step to train one model.The authors report that this global model can still capture temporal dynamics at each depth separately.
  • 2.3 Machine learning model for sequential data: The area-depth profile supplies additional information for energy constraints but is omitted from input features because it is fixed for the target lake.

3 OUR PROPOSED PGRNN METHOD

The proposed PGRNN combines recurrent modeling with explicit lake-energy conservation and uses pre-training to address long-term dependencies and limited observations. Its energy flow links predicted temperatures and heat fluxes while constraining predictions toward physical consistency.

  • 3 OUR PROPOSED PGRNN METHOD: PGRNN combines an LSTM for temperature dynamics, an energy-conservation flow, and pre-training for limited-data learning.
  • 3.1 Recurrent Neural Networks and Long-Short Term Memory Networks: Standard RNNs model temporal transitions but can lose connections to long histories, making long-term environmental patterns difficult to capture.
  • 3.1 Recurrent Neural Networks and Long-Short Term Memory Networks: LSTM addresses long-term dependencies with a cell state that preserves information from the past through gated updates.
  • 3.1 Recurrent Neural Networks and Long-Short Term Memory Networks: The model trains on observed temperature pairs while recurrent transitions continue across time steps without observations.Thus, input drivers at unobserved time steps can still contribute to learning temporal patterns.
  • 3.2 Energy conservation over time: Energy conservation relates changes in lake thermal energy to the balance between incoming and outgoing heat fluxes.Thermal energy is computed from predicted temperatures across depths, water properties, layer areas, and layer thickness.
  • 3.2 Energy conservation over time: PGRNN integrates energy flow into the recurrent process so predicted temperature changes conform to energy conservation.The energy flow explicitly captures heat fluxes transferred between time steps rather than relying only on implicit recurrent representations.
  • 3.2 Energy conservation over time: Energy conservation is applied only during ice-free periods because ice and snow alter reflectance and energy-loss dynamics and are outside this study’s scope.
  • 3.2 Energy conservation over time: The energy-flow procedure requires no true observations, enabling its use in systems with few labeled data points.The conservation loss is combined with the standard recurrent loss, with λEC controlling their balance.

4 EXPERIMENTS

Experiments evaluate recurrent models, energy conservation, and physics-based pre-training on sparse and irregular Lake Mendota observations. The reported evaluations show reduced prediction error, improved physical consistency, and useful accuracy with few observations.

  • 4 EXPERIMENTS: The experiments first test whether an LSTM captures lake dynamics, then evaluate energy conservation and pre-training.
  • 4 EXPERIMENTS: Energy conservation reduces prediction error while maintaining physical consistency in the evaluated model.
  • 4 EXPERIMENTS: Pre-training on simulated data from a physically different lake reduces the observations needed to train a good-quality target-lake model.The source and target lakes differ in geometry, clarity, and climate conditions.
  • 4.1 Dataset: The dataset contains 13,158 observations from Lake Mendota, with measurements distributed irregularly across dates and depths.
  • 4.1 Dataset: Seven continuously available daily drivers describe meteorological conditions, including radiation, air temperature, humidity, wind, freezing, and snow.
  • 4.1 Dataset: Training uses 8,037 observations from two time intervals, while testing uses 5,121 observations from the intervening period.
  • 4 EXPERIMENTS: The energy-loss threshold is 24 W/m2 and λEC is 0.01, selected to balance supervised training loss against energy conservation.
  • 4 EXPERIMENTS: Because the conservation term omits minor physical processes, increasing λEC too much can enforce approximate relationships and produce sub-optimal performance.

4.3 Performance: prediction accuracy and energy consistency

The study evaluates prediction accuracy and energy consistency for RNN, RNNEC, and GLM models under varying observed-data availability. RNNEC generally improves accuracy and energy conservation, especially when observations are limited, although stronger conservation constraints can reduce accuracy on unseen data.

  • Prediction accuracy: RNNEC consistently outperforms RNN, with the largest gap occurring when only 0.2% or 2% of observed data are used.With plentiful observations, RNN achieves performance similar to RNNEC.
  • Prediction accuracy: Both RNN and RNNEC approach their best performance with more than 20% observed data.
  • Prediction accuracy: RNNEC using 20% observed data outperforms the fully calibrated GLM using 100% observed data.
  • Energy consistency: RNNEC better matches energy fluxes with lake energy change, whereas RNN produces a large energy inconsistency.The energy inconsistency is measured from the difference between lake energy change and net energy flux gain.
  • Energy consistency: Increasing λEC reduces RNNEC's energy inconsistency, but strict compliance with the simplified conservation term can reduce accuracy on unseen data.The conservation formula captures only a subset of physical processes and omits certain minor processes.
  • Prediction accuracy: RNNEC achieves lower RMSE than RNN, indicating that the more physically realistic model can also improve prediction accuracy.

4.4 Leveraging the knowledge hidden in physics-based model via pre-training

The paper pre-trains RNNEC on physically consistent GLM simulations and then fine-tunes it with observed data. Pre-training improves accuracy substantially with limited observations while transferring the physics-based model's learned temperature dynamics.

  • Pre-training approach: Pre-training uses GLM simulations to provide synthetic realizations of physical phenomena before fine-tuning with observed data.The approach is intended to leverage complex physical knowledge while reducing dependence on labeled observations.
  • Pre-training approach: Generic GLM simulations have RMSE=2.950 against Lake Mendota observations, yet they still provide useful pre-training data.The experiments use generic, uncalibrated GLM parameters that produce large errors relative to observations.
  • Pre-training results: Pre-training significantly improves RNNEC performance, with larger gains when only small amounts of observed data are available.
  • Pre-training results: Pre-trained models converge in 30–50 epochs, compared with 150–200 epochs for RNN and RNNEC.The authors interpret this as evidence that pre-training provides a better initialization for learning a model.
  • Fine-tuning: Without fine-tuning, RNNEC,p,0 produces predictions similar to generic GLM because it is pretrained to emulate that model.Fine-tuning with 2% observed data substantially closes the gap between RNNEC,p,0 and true observations.

4.5 The RMSE profile across depths and seasons

RMSE varies across depth and season, reflecting challenges in modeling lake stratification. RNNEC,p is especially competitive with calibrated GLM at intermediate and lower depths and during warmer seasons.

  • Error across depths: At shallow depths below 6 m, RNNEC,p performs similarly to generic GLM but has larger errors than calibrated GLM.
  • Error across depths: Generic GLM has much larger errors than RNNEC,p below 6 m, especially at intermediate depths between 6 m and 16 m.
  • Error across depths: Calibrated GLM reduces middle-depth errors relative to generic GLM, but RNNEC,p still has lower errors at lower depths.Calibration helps locate the temperature transition in the specific lake, while stratification dynamics remain difficult to formulate physically.
  • Error across depths: Machine-learning models perform better at intermediate and lower depths by learning patterns without explicit stratification assumptions.
  • Error across seasons: RNNEC,p and calibrated GLM have similar errors in spring, while RNNEC,p outperforms calibrated GLM by a considerable margin in summer and fall.Calibrated GLM improves over RNNEC,p during winter.
  • Error across seasons: GLM performs worse than RNNEC,p in warmer seasons, likely because stratification is stronger or changes more rapidly then.The influence of stratification is weaker in spring and winter than in summer and fall.

4.6 Pre-trained ML model vs. its teacher

The pre-trained RNNEC model does not reproduce all of GLM's lower-depth complexity, but this simplification can yield more accurate lake-temperature predictions than its teacher.

  • Teacher comparison: GLM's Lagrangian layer model can introduce artifacts when layers split or combine, increasing variability at lower depths.These artifacts can be overly sensitive for Lake Mendota and increase GLM error.
  • Teacher comparison: RNNEC,p does not fully capture GLM's complexity and instead predicts smoother, often more accurate temperature dynamics.
  • Teacher comparison: Without refinement, RNNEC,p,0 and GLM perform similarly near the surface, while RNNEC,p,0 has much lower RMSE at lower depths.
  • Teacher comparison: The pre-trained model has higher error relative to GLM outputs at 9 m than at the surface, showing that it cannot fully mimic GLM at lower depths.Its lower error against observations indicates that the simpler learned process can be closer to Lake Mendota dynamics than GLM's output.
  • Teacher comparison: At 9 m, RNNEC,p,0 has RMSE 4.752 against true observations, compared with 5.333 for GLM.The comparison uses models pretrained without observation-based refinement.

4.7 Ability to pre-train using lakes that are very different than target lake

Pre-training on synthetic lakes with mismatched geometry, clarity, or climate drivers can still yield models that adapt to Lake Mendota using very little observed data. Geometry affects transfer most strongly, while fine-tuning substantially reduces errors from mismatched conditions.

  • Experimental setup: Pre-training uses GLM simulations from synthetic lakes with varying geometry, clarity, and climate conditions, followed by fine-tuning on Lake Mendota observations.The experiments test cone, barrel, and martini geometries; normal, dark, and clear water clarity; and Florida climate drivers.
  • Geometry transfer: The cone-pre-trained model adapts well to Lake Mendota even with no observed data, whereas barrel- and martini-pre-trained models initially have much larger errors.The cone geometry is closer to Lake Mendota’s true geometry than the barrel and martini geometries.
  • Geometry transfer: 2% observed data significantly reduces errors for models pre-trained with incorrect geometric structures.The models recover to reasonable performance after fine-tuning despite initially large errors.
  • Geometry transfer: RMSE is 2.469 for cone, 5.340 for martini, and 3.239 for barrel pre-trained models before fine-tuning.These errors are lower than those of their corresponding GLM teachers, indicating lower sensitivity to geometric mismatch.
  • Clarity transfer: Models pre-trained under dark or clear clarity adapt well to Lake Mendota, achieving similar performance across clarity levels with only 0.2% observations.Without fine-tuning, dark clarity produces higher error at depths 6m-12m, where temperature changes most rapidly; 2% data largely removes this difference.
  • Climate transfer: Florida-climate pre-training performs poorly when directly applied to Lake Mendota, with RMSE 9.106, but performance improves substantially after fine-tuning with 2% observations.The source and target climates differ substantially in temperature.

5 RELATED WORK

Prior hybrid and residual approaches combine physics-based and machine-learning models, but differ in whether they enforce internal physical constraints or support synthetic-data pre-training. The PGRNN framework is presented as addressing both limitations while producing physically consistent outputs.

  • Residual modeling: Residual modeling trains machine-learning models to predict errors made by physics-based models and then corrects their outputs.These approaches generally require observations to compute the physics-model errors.
  • Residual modeling: Residual approaches cannot enforce physics-based constraints because they model errors rather than physical quantities.This limits their ability to impose constraints such as energy conservation.
  • Hybrid models: A hybrid model feeds physics-model outputs and driving inputs into machine learning, allowing physics constraints on the physical-quantity output and label-free learning.The machine-learning component can retain physics-model behavior where it performs well and correct mistakes elsewhere.
  • Hybrid models: These hybrid approaches still require observations during training and therefore cannot initialize the machine-learning model using only synthetic physics-model outputs.Synthetic outputs are available without observation labels, but the cited approaches do not use them for initialization alone.
  • Physics-model emulation: Physics-model emulators can avoid training-data scarcity through synthetic outputs, but generally cannot correct errors caused by missing physics or incorrect parameterization.Their expected performance is bounded by the physics models generating the training data.
  • PGRNN positioning: PGRNN combines physics-based models with deep learning to produce physically consistent emulators that may be more robust and generalizable out of sample.The framework can also guide surrogate models described in other physics–machine-learning approaches.

6 CONCLUSIONS

The conclusions present PGRNN as a framework that transfers knowledge from physics-based simulations into machine-learning models while incorporating physical laws. In lake-temperature experiments, it achieves strong prediction with sparse observations and is positioned for broader scientific and engineering applications.

  • Contributions: PGRNN provides a framework for modeling spatial and temporal physical processes while incorporating energy conservation.The approach adds states computed from physical equations, enabling richer physical constraints in the loss.
  • Empirical conclusions: Using simulated data from a generic physics-based model, PGRNN achieves high prediction performance with fewer observations than a calibrated physics-based model.The comparison is made against a parameterized physics-based model calibrated using many observations.
  • Interpretation: PGRNN leverages physics-based model strengths while filling knowledge gaps with data-driven predictive models.The framework is described as transferring knowledge from physical processes to machine-learning models.
  • Scope: The framework can incorporate mass conservation and is applicable to domains including hydrology and computational fluid dynamics.The paper also identifies environmental and engineering applications such as nutrient exchange, crop production, and vehicle refueling design.
  • Scope: PGRNN’s ability to create accurate models from few observations is presented as relevant to applications with many variables.The stated framework components include physical-law terms in the loss and simulation-data pre-training.
  • Limitations and future work: Future research must determine which dynamical-system models benefit from the approach and whether architectural enhancements are needed.The conclusion leaves the effective system types and required architectural changes unresolved.

A ENERGY CONSERVATION

The lake energy balance accounts for incoming and outgoing heat fluxes, with detailed terms derived from environmental drivers and modeled surface temperature. The implementation simplifies the balance by omitting smaller fluxes and fixing albedo values.

  • Energy balance: The detailed energy balance defines ΔUt as the change in lake energy and separates net incoming and outgoing fluxes.Fin combines absorbed short- and long-wave radiation, while Fout combines back radiation, evaporation, and sensible heat.
  • Energy balance: The implementation sets short-wave and long-wave albedo to 0.07 and 0.03, respectively, with all energy components measured in Wm−2.These are described as generally accepted lake values from previous scientific studies.
  • Assumptions: Smaller terms such as sediment heat flux and advected energy from surface inflows are ignored in the energy-balance implementation.This is an explicit modeling simplification of the lake energy budget.
  • Heat fluxes: Incoming lake heat fluxes include terrestrial long-wave and short-wave radiation, whose drivers include local conditions, latitude, season, and cloud cover.Both radiation components are included in the input drivers X.
  • Heat fluxes: Outgoing energy fluxes are represented by back radiation, sensible heat flux, and latent evaporative heat flux.The fluxes E, H, and RLWout are estimated separately from input drivers and modeled quantities.
  • Heat fluxes: Sensible and latent heat fluxes use bulk aerodynamic coefficients estimated from Hicks’ collection of ocean and lake data.CH describes sensible heat transfer, while CE describes latent heat transfer.

B DISTRIBUTION OF TRAINING DATA

Training uses a randomly selected subset containing 2% of observed data, distributed across different lake depths and dates.

  • Training-data distribution: The selected observations are distributed across different depths and different dates.The data are randomly selected across both dimensions.
  • Training-data distribution: Figure 14 visualizes the distribution of the randomly selected observed training data.The figure focuses on how the 2% sample is spread across depths and dates.
  • Training-data distribution: 2% of observed data are used for training.The figure describes the observed-data fraction used in the training set.
Loading 2001.11086v3…