Integrating Scientific Knowledge with Machine Learning for Engineering and Environmental Systems

Jared Willard; Xiaowei Jia; Shaoming Xu; Michael Steinbach; Vipin Kumar

Source-linked AI summary

Integrating Scientific Knowledge with Machine Learning for Engineering and Environmental Systems

Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, Vipin Kumar

arXiv:2003.04919v6Mar 10, 2020physics.comp-phcs.LGstat.ML

TL;DR

Engineering and environmental systems often require both mechanistic models and machine learning because each has important limitations alone. This survey organizes integration strategies and application objectives into a taxonomy, identifying gaps and opportunities for cross-disciplinary research.

Problem
Black-box ML can require large datasets, produce physically inconsistent outputs, and generalize poorly, while mechanistic models approximate reality and may be limited by incomplete process knowledge.
Method
The survey reviews application-centric objectives and classes of physics-guided ML and hybrid physics-ML methodologies, then organizes them in a taxonomy.
Results
The taxonomy covers scientific-knowledge integration approaches across engineering and environmental objectives and reveals sparse areas and opportunities for future work.
Takeaways & Limitations
Integrating scientific knowledge with ML provides a structured basis for combining complementary modeling strengths across diverse scientific applications.
Takeaways & Limitations
The survey focuses primarily on engineering and environmental systems traditionally addressed with mechanistic models, although related ideas apply more broadly.

Abstract

from arXiv · show

There is a growing consensus that solutions to complex science and engineering problems require novel methodologies that are able to integrate traditional physics-based modeling approaches with state-of-the-art machine learning (ML) techniques. This paper provides a structured overview of such techniques. Application-centric objective areas for which these approaches have been applied are summarized, and then classes of methodologies used to construct physics-guided ML models and hybrid physics-ML frameworks are described. We then provide a taxonomy of these existing techniques, which uncovers knowledge gaps and potential crossovers of methods between disciplines that can serve as ideas for future research.

1 INTRODUCTION

The survey examines how scientific knowledge and machine learning can be integrated for engineering and environmental systems, where neither mechanistic models nor ML alone is sufficient. It reviews application objectives, integration methods, and a taxonomy highlighting research gaps and cross-disciplinary opportunities.

ML is promising for poorly understood or computationally expensive scientific processes, but black-box models require large datasets and may lack physical consistency and generalizability.
The survey frames physics-ML integration as a continuum combining complementary strengths of mechanistic modeling and ML.
The literature has expanded rapidly across scientific disciplines, with publications on “physics-informed ML” increasing from 2 in 2017 to 63 in 2020.
The survey interprets “physics-guided” and “physics” broadly as scientific-knowledge-guided approaches.
It focuses on mechanistic models, theories, and laws in environmental and engineering systems and proposes a taxonomy spanning objectives and computational methods.

2 APPLICATION-CENTRIC OBJECTIVES OF PHYSICS-ML INTEGRATION

Physics-ML integration targets improved prediction, resolution, computational efficiency, knowledge discovery, data generation, and uncertainty quantification across engineering and environmental applications. These objectives address limitations of approximate physics models, expensive simulations, insufficient resolution, and data scarcity.

Improving physical models: Physics-ML can improve mechanistic-model effectiveness and predictive accuracy when processes are poorly understood and observations are available.
Resource efficiency: ML can serve as a faster surrogate for downscaling, parameterization, reduced-order modeling, and PDE solution when physical simulations are too slow or coarse.
Knowledge discovery: Inverse modeling infers causal physical parameters from system outputs, while equation discovery seeks explicit symbolic representations of governing processes.
Data generation and uncertainty: Data generation creates synthetic input or output distributions for data-limited settings, while UQ models output uncertainty arising from other model components.
Motivation: Physics-based models can suffer from approximation bias, parameter-estimation errors, and heterogeneity across space and time.
Downscaling: Downscaling uses statistical mappings or dynamical simulations to predict finer-resolution variables, with ML surrogates reducing the cost of complex high-resolution models.

2.3 Parameterization

Parameterization replaces unresolved dynamical processes with simplified representations, and physics-ML methods seek to learn these representations while improving robustness and efficiency. Related ML-based reduced-order models use surrogates, learned mappings, residual correction, and Koopman-operator approximations.

Parameterization: Parameterization represents missing physics by replacing complex dynamical processes with simplified physical approximations expressed as static parameters.
Parameterization: ML can learn new parameterizations directly from observations and high-resolution simulations when processes are too complex to represent physically.
Physics integration: Integrating physics into parameterization models may improve robustness and generalizability to unseen scenarios while reducing training-data requirements.
Reduced-order models: ML-based reduced-order models can act as surrogates for full-order models, approximate reduction mappings, or learn residuals between reduced models and observations.
Reduced-order models: Deep learning has been used to approximate dominant Koopman-operator modes, with physics-based knowledge potentially improving generalizability and interpretability.
Reduced-order models: Physics principles could reduce the search space for reduced-order models, enabling more robust training with less data in some scenarios.

2.5 Forward Solving Partial Differential Equations

ML-based methods can address computationally expensive forward simulation and inverse inference in systems governed by PDEs or mechanistic models. The survey also connects equation discovery and physics-guided constraints to more interpretable and potentially more robust modeling.

Forward solving: When direct numerical PDE solutions are prohibitively expensive, ML models can act as surrogates to reduce computation time.
Forward and inverse problems: Forward modeling predicts system states or outputs from physical parameters, whereas inverse modeling infers intrinsic parameters or inputs from possibly noisy outputs.
Inverse modeling: Inverse problems can reveal valuable information that cannot be observed directly, including intrinsic physical parameters in engineering and environmental systems.
Inverse modeling: ML surrogates are attractive for inverse problems because they model high-dimensional phenomena and execute faster than most physical simulations.
Inverse modeling: Deep learning has been applied to inverse problems in areas including remote sensing, hydrology, photonics, medical imaging, and computerized tomography.
Inverse design: Physics-based constraints and material-property stopping conditions can guide ML optimization for inverse material design.
Equation discovery: Known governing equations support forecasting, control, stability analysis, bifurcation analysis, and generalization outside the training domain.

2.8 Data Generation

Scientific ML approaches address data-generation and uncertainty-quantification needs by combining generative or surrogate models with physical knowledge. These methods seek faster, more scalable simulations while improving physical consistency and reducing sample or computational demands.

Data Generation: Physics-based data generation is time consuming and limited by what physical models can produce, motivating generative ML methods that learn data distributions and generate novel scientific data.Generative approaches can create virtual simulations under specified conditions, including chemical compounds with desired characteristics.
Data Generation: GANs can reduce computation time and reproduce complex nonlinear phenomena when generating scientific data such as heat-conduction and fluid-flow simulations.The cited example uses conditional GANs trained on observations to simulate these processes without relying solely on knowledge from physical models.
Data Generation: Physics constraints can improve generative models by enforcing conservation laws, energy spectra, or morphology distributions in generated scientific data.A morphology constraint for artificial materials forces generated samples to match authentic morphology distributions, reducing the material design space.
Uncertainty Quantification: Uncertainty quantification requires modeling the full conditional distribution p(y|x), rather than only producing point predictions, to characterize quantiles, skew, and sensitivity.This distributional view supports analyses of predictions near unacceptable outcomes and the sensitivity of outputs to input features.
Uncertainty Quantification: Traditional Monte Carlo uncertainty quantification is often infeasible because convergence requires many forward-model evaluations, while Gaussian Processes scale as O(N^3) with N data points.Reduced models and ML surrogates are used to accelerate evaluations, but Gaussian Processes and reduced models do not scale well to higher dimensions or larger datasets.
Uncertainty Quantification: Deep-learning surrogates support faster and more scalable uncertainty quantification through dropout, Bayesian neural networks, or ensembles, while physics integration can improve physical consistency.Bayesian neural networks may incur high computation times and depend strongly on reliable priors; physics constraints may also reduce data needs.

3 PHYSICS-ML METHODS

Physics-ML methods integrate scientific principles into machine-learning models through four methodological classes. These approaches pursue prediction performance, resource efficiency, and improved sample efficiency, with method choice depending partly on available mechanistic models and computational objectives.

Method Classes: The survey categorizes physics-ML methods into physics-guided loss functions, initialization, architecture design, and hybrid modeling.The four classes reflect different ways of incorporating physical principles into ML models.
Method Selection: Method selection depends on the availability and performance of existing mechanistic models and on the computational objectives of the application.The survey compares method requirements and the computational objectives they achieve.
Computational Objectives: Physics-ML computational objectives include better prediction performance, improved resource efficiency, and greater sample efficiency.The supplied discussion links physical consistency and more effective use of observations to these objectives.
Computational Objectives: Enforcing physical consistency can reduce the solution search space, while improved sample efficiency can increase prediction performance by extracting more value from each observation.These mechanisms distinguish the objectives pursued by physics-ML methods from those of traditional ML methods.

3.1 Physics-Guided Loss Function

Physics-guided loss functions incorporate scientific constraints into ML training to improve physical consistency, data efficiency, and performance across environmental and engineering objectives. The survey describes applications including prediction, inverse modeling, uncertainty quantification, generative modeling, PDE solving, and governing-equation discovery.

Physics-based loss terms augment standard supervised and complexity losses with a weighted physical-consistency term.The physical term is controlled by hyper-parameter γ, selected with other ML hyperparameters using validation or nested cross-validation.
Physical constraints can reduce the solution search space and regularize learning with unlabeled data, but they do not guarantee physical consistency or generalizability.The survey characterizes physics-guided loss as a weak constraint.
Replacing or improving over physical models: Energy-conservation penalties improve lake-temperature modeling by constraining thermal-energy changes to match net thermodynamic fluxes.PGRNN models include physics-derived energy variables alongside recurrent neural-network computations.
Solving PDEs and discovering governing equations: Loss-function constraints enable data-efficient PDE approximation and improve time-accurate governing-equation discovery while reducing convergence time in selected systems.Applications include Burgers and Schrödinger equations and a Lorenz-system echo state network.
Application objectives: Physics-guided losses support inverse modeling, parameterization, downscaling, and uncertainty quantification by enforcing consistency with governing physical relationships.Examples include sparse-data force and obstacle inference, atomic-energy parameterization, PDE-constrained super resolution, and PDE-structured probabilistic models.
Generative models: Physics-based losses improve or extend generative and dynamical-system models, including GAN solutions for stochastic elliptic PDEs up to 30 dimensions and stability-regularized prediction.Reported applications also reduce data or training requirements and improve generalization in selected settings.

3.2 Physics-Guided Initialization

Physics-guided initialization uses contextual or physical knowledge to provide better starting states for ML training. The survey covers simulation-based pre-training, transfer learning, Gaussian-process adaptation, and self-supervised representations.

Physics-informed initialization can accelerate training and reduce sample requirements by replacing random starting states with knowledge-informed parameters.Poor initialization can anchor deep neural networks in local minima.
Transfer learning: Transfer learning adapts a model pre-trained on a related task, including pre-training ML models on simulated data from physics-based models.The pre-trained model is fine-tuned with limited data for the desired task.
Simulation-based pre-training: Lake-temperature PGRNNs pre-trained on physics-model simulations required less training data for quality models, even when the simulator used incorrect parameters.The models were fine-tuned with little observed data.
Simulation-based pre-training: Simulation-based pre-training reduced real-world data needs in robotics and autonomous-vehicle training, including a factor-of-50 reduction for object grasping.Robotics simulations also supplied sufficient images for object localization without real-world data.
Other initialization strategies: Physics-guided initialization has also been applied to chemical-process Gaussian-process models and self-supervised learning based on pseudo-labels from physical variables.Transferred Gaussian-process models use scale-bias correction, while self-supervised pretext tasks learn representations related to the target prediction task.

3.3 Physics-Guided Design of Architecture

Physics-guided architectures embed domain structure directly into ML models through physical variables, invariances, symmetries, operators, and specialized layers. These designs aim to improve physical consistency, generalizability, accuracy, computational search, and interpretability.

Physics-guided architectures encode domain characteristics directly into ML structures rather than relying only on losses or initialization.Examples include domain-informed convolutions, discriminators, and structures based on physical characteristics.
Intermediate Physical Variables: Physically meaningful intermediate variables can produce physically consistent predictions and expose hidden representations that domain scientists can interpret.Applications include lake-temperature modeling and drag-force prediction for particle suspensions.
Encoding invariances and symmetries: Symmetry- and invariance-aware architectures constrain predictions so equivalent physical configurations receive consistent outputs.Examples include rotational invariance for turbulence models and translational, rotational, scale, and uniform-motion symmetries in spatiotemporal networks.
Encoding invariances and symmetries: Physics-informed architecture design can reduce the solution search space, which is especially relevant when discovering governing equations with exponentially many candidate terms and operators.The passage describes this research direction as still being in its infancy.
Specialized architectures include atom-shared networks for energy invariance, PDE-guided CNN variants, Fourier neural operators, and Hamiltonian neural networks.Fourier neural operators were reported to improve accuracy over earlier ML-based PDE solvers and solve families of PDEs; Hamiltonian networks learn and respect energy conservation in simple systems.
Physics-guided Gaussian process regression: Physics can also enter Gaussian processes through differential-equation-informed kernels and matrix-valued kernels encoding physical symmetries.This approach retains Gaussian-process benefits such as effectiveness with small datasets and uncertainty measurements.

3.4 Hybrid Physics-ML Models

Hybrid physics-ML models make physical and learned components operate together, either by correcting physical-model errors, using physical outputs as ML inputs, replacing components, or combining predictions adaptively. These frameworks extend residual modeling to more integrated forms of physical-model augmentation.

Hybrid physics-ML models combine physical and ML components simultaneously, sometimes replacing the mechanistic model or one of its subcomponents.They are also described as ML-enhanced physical models.
Residual modeling: Residual modeling trains ML to predict a physics-based model’s errors and adds the learned residual to the physical prediction.The approach corrects physical-model bias relative to observations but cannot enforce physics-based constraints by itself.
Residual modeling: Residual models can correct reduced-order-model truncation, reduce computational cost and time-discretization error, or generate corrective source terms inside governing equations.The corrective-source-term approach modifies the governing equation rather than only its output.
Output of physical model as input to ML model: A hybrid model can feed simulated physical outputs and the original input drivers jointly into an ML model to produce the final prediction.This structure was applied to lake-temperature prediction.
Other hybrids replace poorly modeled physical components or predict intermediate quantities, such as turbulence-closure variables missing from RANS models.Parameterization models can also be viewed as hybrid modeling, although most use black-box ML.
Adaptive hybrid frameworks weight physical and ML predictions according to the prediction circumstance, such as process scale or estimator quality.Examples combine physics for long-range interactions with ML for short-range stochastic interactions.
ML informing or augmenting physics-model for inverse modeling: For inverse modeling, hybrid systems may perform direct physics-based inversion first and then use deep learning to refine the result.Applications include computed tomography and CNN priors embedded in physics-based MRI forward models.

3.5 Requirements and Benefits from Different Physics-ML Methodologies

Physics-ML methodologies differ in their knowledge, data, and runtime requirements, so suitability depends on the application and available resources. Combining methods can address efficiency while retaining predictive performance, sample efficiency, and interpretability goals.

The taxonomy summarizes requirements and possible benefits across four physics-ML methodology types.The methods are intended to support different application contexts and resource constraints.
Residual modeling is comparatively simple because it requires no process-based knowledge beyond an operational mechanistic model at runtime.Physics-guided loss functions require domain expertise to select loss terms, with cross-validation recommended for weighting them.
Only some methodologies apply to a given objective: hybrid methods are unsuitable when retaining a solver prevents computational reduction, and residual models require a reference physical model.Data-generation objectives likewise do not fit residual modeling because they simulate distributions rather than improve physical-model predictions.
Physics-guided losses, architectures, and initializations can be combined within one ML model or with a hybrid physics-ML framework.The survey cites examples combining losses with initialization and losses with hybrid modeling.
The overall goal is to use fewer computational resources while maintaining predictive performance, sample efficiency, and interpretability relative to traditional ML.The survey identifies PDE-solving methods as a setting where physics-ML may be more computationally efficient than direct numerical approaches.

4 AREAS OF CURRENT WORK AND POSSIBILITIES FOR CROSS-FERTILIZATION

The survey’s taxonomy exposes sparse or empty combinations of objectives and methods, revealing opportunities to transfer physics-ML ideas across disciplines. It also identifies data assimilation and continual state updating as promising directions beyond the taxonomy.

The taxonomy organizes application-centric objectives and methods while revealing knowledge gaps and potential crossovers between disciplines.These gaps and crossovers are presented as ideas for future research.
Physics-guided methods developed for aquatic sciences may transfer to other applications with imperfect mechanistic models, while loss-based methods can extend across objectives.The survey gives examples involving PDEs, inverse modeling, and cross-application transfer.
Sparse or empty taxonomy cells indicate opportunities to apply existing methods to underexplored objectives such as physics-guided climate and weather parameterization.Super-resolution ideas are beginning to support downscaling, but many implementations do not yet incorporate physics.
Data assimilation offers an additional direction in which ML forecasts future events and continuously updates physical-model states.Examples include predicting physical-model parameters from past time series data.

5 CONCLUDING REMARKS

The survey argues that integrating ML and physics can address pressing environmental and physical modeling problems while enabling cross-disciplinary exchange. Its taxonomy identifies research gaps, broader applications, and transferable methods across scientific communities.

Combining ML and physics is presented as valuable for future scientific modeling of pressing environmental and physical problems.The survey links this direction to current sensor-data growth and advances in ML methods.
The taxonomy helps researchers position their methods and identify relationships with methods used for other application objectives.This organization is intended to support cross-pollination among diverse research communities.
Physical constraints in loss functions can support fairness in predictive models and realism in GAN-generated data.These examples illustrate how physics-ML techniques may benefit the broader ML community beyond engineering and environmental modeling.
Novel architectures can incorporate prior domain information beyond typical Bayesian frameworks and may improve interpretability.
The survey primarily targets engineering and environmental systems traditionally addressed with mechanistic models, while related integration efforts span control, computational chemistry, physics, hydrology, geostatistics, and neuroscience.The authors identify potential for ideas to cross over among these efforts.