Combining physics-based and data-driven models: advancing the frontiers of research with Scientific Machine Learning

1 Introduction

• Physics-based modelling translates physical principles into mathematical models, approximates them numerically, and verifies and validates the resulting computational solutions.The paper describes this as a widely used strategy for complex problems.Source: §1 · p. 3
• Cheap computing, large datasets, and powerful learning algorithms have accelerated neural-network-based machine learning.Data-driven models learn input–output relations from training data without inferring causality.Source: §1 · p. 3
• SciML combines physics-based models and machine-learning algorithms, using physical knowledge to improve learning and data-driven patterns to enhance digital models.The framework includes physics-informed loss regularization, data augmentation, and learning constitutive laws or accelerating computation.Source: §1 · p. 4
• The paper surveys SciML strategies for PDE problems and uses the Integrated Heart Model as a challenging test bed for cardiac-function simulation.Cardiac simulation couples multiple physiological processes and requires supercomputing for its large system of nonlinear differential equations.Source: §1 · pp. 4–5
• Cardiac models need patient geometry, initial and boundary conditions, and material parameters, but some inputs are seldom available and others are difficult to recover accurately or noisy.The paper also notes that more complex models require more parameters; data assimilation, parameter identification, and uncertainty quantification can address data scarcity or inaccuracy.Source: §1 · p. 4

2 Digital models

• Digital models comprise numerical approximation and computer implementation, following problem understanding and mathematical modelling.Their design requires understanding the mathematical solution and the original problem's phenomenology.Source: §2 · p. 5

2.1 Mathematical models

• Mathematical models encode relations between data and solutions, simplifying the physical problem; the resulting difference is intrinsic model error.The model error is the distance between the physical and mathematical solutions, reflecting both simplifying assumptions and how well the data represent physical quantities.Source: §2.1 · pp. 6–7

2.2 Numerical models

• Because exact solutions may be infinite-dimensional and computers cannot compute exact derivatives, integrals, or limits, numerical models discretize the problem into finite-dimensional approximations.Smaller spatial mesh sizes and time steps typically improve accuracy but increase computational cost.Source: §2.2 · pp. 7–8
• Larger discretizations can produce ill-conditioned systems that impair accuracy or slow iterative solvers, motivating preconditioners that improve conditioning and scalability.No single optimal preconditioner works for all linear systems.Source: §2.2 · p. 8
• When monolithic multiphysics systems are too large, operator splitting reduces complexity and reuses solvers but may reduce accuracy, while domain decomposition treats physics in distinct subdomains.Splitting can also accommodate different spatial and temporal scales; domain decomposition permits non-conforming discretizations at subdomain interfaces.Source: §2.2 · pp. 8–9
• Convergence requires consistency and stability; a-priori analysis predicts error decay as discretization is refined, while a-posteriori analysis uses residuals to assess tolerance and guide adaptation.A-priori estimates generally do not quantify error accurately for a fixed discretization.Source: §2.2 · p. 9
• Physical parameters can be difficult to measure; identification fits model outputs to observations, while uncertainty analysis distinguishes epistemic from aleatory uncertainty and examines input-output sensitivity.Parameter identification may require solving many instances of the same model.Source: §2.2 · p. 10
• Reduced order models build a low-dimensional basis from solution snapshots and project the equations onto it, making new-parameter solutions cheap after costly offline preparation.For nonlinear or non-affine parameter dependence, hyper-reduction strategies are needed.Source: §2.2 · pp. 10–11
• Computer implementation adds finite-arithmetic error ea to numerical error eh, producing computational error ec between the computed solution uc and the exact solution.The overall error e between the physical solution uph and uc combines modelling error em with ec.Source: §2.2 · p. 11
• Digital modelling seeks to keep ec small and controllable while minimizing computation time and memory use.Reliability is assessed by estimating ec in a suitable norm from problem data and discretisation parameters against a predetermined threshold.Source: §2.2 · p. 11
• Adaptive computational models use feedback from computed results to adjust discretisation parameters and improve solution quality.If algorithmic error ea is controllable within the desired tolerance, ec behaves like eh and tends to zero as h tends to zero.Source: §2.2 · p. 11

• Data-driven models use measurements, imaging, or other experiments with machine-learning algorithms, typically artificial neural networks, to obtain solutions.Source: §3 · p. 11
• Modern data-driven models use machine learning rather than analytical approaches to draw conclusions from data, unlike historical examples such as Kepler and Newton’s work on planetary orbits.Source: §3 · p. 11
• The paper frames AI as a fourth pillar alongside theory, experiments for validation, and numerical simulations in digital modelling.Source: §3 · p. 11

• AI is described as techniques that enable computers to mimic human intelligence, including learning, reasoning, problem-solving, perception, and decision-making.Source: §3.1 · p. 11
• AI applications span areas including healthcare, environment, energy, manufacturing, finance, transportation, and security.Source: §3.1 · p. 11
• The section associates AI’s growing scientific interest with increasingly powerful, relatively cheap computing hardware capable of rapid heavy computation.Source: §3.1 · p. 12
• The section also identifies abundant, low-cost data and powerful machine-learning algorithms, especially deep neural networks, among the factors behind AI’s growth.Source: §3.1 · p. 12
• The three software tasks are discriminative classification, generative production of new information, and reward maximization for decision-making agents.Source: §3.1 · p. 12
• Generative AI’s recent development is linked in the section to large language models such as GPT-3, GPT-4, T5, and Gemini.Source: §3.1 · p. 12
• For discriminative tasks, statistical learning trains and tests on datasets from a prescribed distribution, with learnability and computational complexity as central mathematical issues.Source: §3.1 · p. 12
• Online learning updates models as a continuous stream of training data arrives, making the statistical train-test paradigm unsuitable; its theory is connected to repeated games.Source: §3.1 · p. 12

3.2 Machine Learning

• ML works less well where large datasets and contextual validation are unavailable, including settings with sparse data, no training data, or high-consequence interpretability needs.The section identifies medicine as a setting where data acquisition may be invasive or extremely costly.Source: §3.2 · p. 13
• A standard learning workflow fits candidate models to training examples by optimizing a loss, then evaluates a performance metric on separate test data.The metric generally monitors performance, while the loss guides training; they need not be the same function.Source: §3.2 · pp. 13–14
• Linear least-squares regression can reduce training to solving a linear system, but a degree-one model imposes a strong prior on the input–output relationship.The section contrasts this computational efficiency with richer model classes that represent more complex relationships but add algorithmic and computational challenges.Source: §3.2 · p. 15
• Typical ML problems involve very large datasets, many free parameters, and nonlinear models that require automatic differentiation and iterative optimization.The section names gradient descent, stochastic gradient descent, Newton, and BFGS among possible optimization methods.Source: §3.2 · p. 15

3.2.1 Machine Learning Tasks

• ML tasks include density estimation, dimensionality reduction, regression, classification, clustering, and association.Examples include anomaly detection, data compression, continuous-value prediction, image labeling, segmentation, and recommendation systems.Source: §3.2.1 · p. 16

3.2.2 Machine Learning Experience

• Supervised learning uses input–output pairs, unsupervised learning uses inputs without known outputs, and reinforcement learning involves an agent learning through actions and rewards.Semi-supervised learning is described as using only a small labeled subset of the training set.Source: §3.2.2 · p. 16

3.2.3 Machine Learning Performance measurement

• MSE, RMSE, and MAE are typically used for regression, whereas cross-entropy is used for classification.Cross-entropy derives from the maximum likelihood estimator for categorical variables.Source: §3.2.3 · p. 17

3.2.4 Machine Learning Models

• Linear models suit regression, while polynomial models are used when data do not fit a linear model and remain linear in their parameters.A linear model has form f(x; w) = Wx + b; for one input, a degree-q polynomial has form f(x; w) = a0 + a1x + ... + aqx^q.Source: §3.2.4 · p. 17
• Decision trees use data-inferred binary decisions for classification and regression but are generally not robust across different datasets; random forests reduce variance and overfitting through bootstrapped trees and averaging.Random forests are described as robust and accurate despite their greater complexity.Source: §3.2.4 · p. 18
• Maximal margin classifiers separate two classes by maximizing the minimum distance to a hyperplane, while SVMs extend this approach to nonlinearly separable classes using kernels.The training points that realize the margin are called support vectors.Source: §3.2.4 · p. 18
• Perceptrons combine weighted inputs, a bias, and an activation function; feed-forward networks compose neuron layers, and their weights and biases are learned by gradient methods using backpropagation.Layer count, neurons per layer, and activation are hyperparameters; deep-network optimization is highly nonlinear and non-convex.Source: §3.2.4 · pp. 19–21
• CNNs use filter-like kernels and are effective for signal processing and computer vision, while spiking neural networks process temporally distinct spikes but still lack solid training methods.The biologically plausible Hodgkin–Huxley spiking model is expensive and infeasible for large-scale simulations.Source: §3.2.4 · p. 22

3.2.5 Setting of supervised learning and error analysis

• Supervised learning seeks a function that predicts targets from inputs; its design depends on the hypothesis space, training set, and optimization algorithm, with approximation, generalization, and optimization errors.The ideal expected-risk minimizer is infeasible to compute directly, so a model is selected and optimized using finite training data.Source: §3.2.5 · pp. 22–23
• Shallow FFNNs with discriminatory activations can approximate continuous functions on compact sets to arbitrary precision, but their required neuron count grows as tolerance shrinks and exponentially with input dimension.For fixed tolerance, greater target-function regularity reduces the required neuron count, while higher input dimension increases it.Source: §3.2.5 · pp. 23–24
• Deep ReLU networks approximate bounded-norm Sobolev functions with architecture sizes that depend on accuracy and regularity, with upper bounds extending Yarotsky’s result to more general Sobolev spaces [260, 93].The stated results bound layer, parameter, and neuron counts while controlling approximation error in specified norms.Source: §3.2.5 · p. 24
• Deep FFNNs do not eliminate the curse of dimensionality, but can represent some functions unavailable to shallow networks and typically approximate more efficiently with fewer parameters [180, 93].With the same parameter count, deep networks can also represent more complex functions than shallow ones.Source: §3.2.5 · p. 25
• Capacity, set by hyperparameters such as polynomial degree or neural-network size, reduces empirical risk as it grows but can increase expected risk and overfitting when excessive.Underfitting reflects a hypothesis space too coarse to fit the input-target relation; overfitting reflects fitting training noise and a larger generalization gap.Source: §3.2.5 · p. 25, Figure 10 · p. 26
• Cross-validation separates training data, which updates network weights and biases, from validation data, which assesses current performance, enabling overfitting monitoring and mitigation.The text recommends stopping when validation error no longer improves; overfitting can also be addressed with more training data, lower capacity, or regularization [86, Ch. 5].Source: §3.2.5 · p. 26
• For binary classification, VC dimension measures hypothesis-space complexity by the largest set of points it can shatter; linear binary classifiers in R2 have VC dimension 3 [242, 167, 180, 91].A set is shattered when the hypothesis space can correctly classify it for every assignment of labels.Source: §3.2.5 · p. 26
• The VC-based generalization bound implies that, with fixed VC dimension, generalization error vanishes as training-set size grows, while with fixed sample size it increases as VC dimension grows.The passage also states that sample size and VC dimension should increase at the same rate to maintain the same gap and guarantee best performance.Source: §3.2.5 · p. 27

Loss minimization and nonconvexity

• Training minimizes empirical loss rather than expected risk, and nonconvex losses can lead descent to a local rather than global minimizer, so practice often settles for a low-loss local solution.The outcome may depend on initialization; finding the global minimizer of a nonconvex loss is typically infeasible or too costly.Source: §3.2.6 · pp. 27–29

Descent directions and learning rates

• Gradient directions have the lowest per-iteration cost but may require more iterations; conjugate-gradient directions typically outperform them, while costlier Newton and quasi-Newton methods provide quadratic and superlinear convergence, respectively.Line search or trust strategies can improve convergence but are costly; for large problems, a constant learning rate may be preferable if calibrated carefully, since rates that are too small or large can slow convergence.Source: §3.2.6 · p. 28

GD vs SGD: costs and convergence rates

• GD is more convenient for moderate training sets, whereas SGD is preferable for large ones: GD converges linearly but each iteration scales with N, while SGD converges sublinearly at per-iteration cost independent of N.When N is large relative to the iteration count, SGD behaves similarly on empirical and expected risk, suggesting it is less susceptible to overfitting than GD.Source: §3.2.6 · p. 31

Minibatch and noise-reduction methods

• Minibatch SGD has costs comparable to SGD while better exploiting GPUs; DSS geometrically increases batch size to achieve linear convergence at SGD complexity under strict convexity, and gradient aggregation reuses stored gradients.Minibatches combine several randomly selected individual gradients, providing a compromise between full-gradient and single-sample updates.Source: §3.2.6 · pp. 31–32

Momentum methods

• With momentum, the descent direction combines the stochastic gradient and previous descent direction; its smoothing parameter prevents gradient descent from slowing in flat loss regions or becoming trapped around local minima.Nesterov momentum computes the loss gradient at a linear combination of the two latest iterations and improves convergence when applied to full gradient descent (Nb = N).Source: §3.2.6 · p. 32

Adaptive learning rates

• Adaptive methods scale learning rates componentwise; AdaGrad can shrink steps too quickly and performs less well on nonconvex problems, while RMSProp discards older gradient history and works better especially on nonconvex functions and deep networks.The scaling strategy takes smaller steps along directions with larger gradient components and larger steps along directions with smaller components.Source: §3.2.6 · p. 33

Adam

• Adam [132] uses exponentially weighted first- and second-order gradient moments with momentum, and typical hyperparameters are β1 = 0.9, β2 = 0.999, ε = 10−8, and η = 10−3.Source: §3.2.6 · p. 33

Second–order and other methods

• Second-order methods capture loss curvature, but often underperform gradient descent in deep-network training because they scale poorly to large networks and encounter many saddle points.High nonlinearity and ill-conditioning can also make first-order methods converge slowly because gradients alone provide limited information about the minimum.Source: §3.2.6 · p. 34
• Stochastic Trust Region Methods [25] use a local quadratic loss model to provide both the displacement direction and step length; under suitable probabilistic assumptions, they find an ε-approximate minimizer in at most O(ε−2) inexact evaluations of the function and its derivatives [25].Source: §3.2.6 · p. 34

3.2.7 Backpropagation

• For a generic input and target, the network output is computed by feed-forward evaluation, with the Euclidean norm assumed for the loss distance.The feed-forward output is f(x̂; w) = a[L].Source: §3.2.7 · p. 35
• Backpropagation computes individual-loss gradients efficiently to determine weights and biases that minimize the loss function.The procedure combines backpropagation with stochastic gradient descent using sampling with replacement and a constant learning rate.Source: §3.2.7 · p. 35
• The chain rule propagates derivatives from the last layer backward, using each layer’s weighted inputs and activation function.The layerwise relations are rewritten as a backward loop for computation.Source: §3.2.7 · p. 35

3.2.8 Penalty–based regularization

• Regularization adds a strictly convex term to the loss, typically penalizing weights rather than biases because large weights may make neurons less reliable on unseen data.The change to backpropagation is described as minor and inexpensive.Source: §3.2.8 · pp. 36–37
• L2 regularization usually outperforms L1 in accuracy, while L1 yields sparse weights that can identify application-relevant features.The sparse solution can indicate inputs or neurons that are meaningless and can be dropped from the network.Source: §3.2.8 · p. 37

3.2.9 Tuning of hyperparameters

• Hyperparameters control network capacity or optimization behavior and are tuned separately from weights and biases, manually or through computationally intensive optimization.Examples include layer and neuron counts, learning rate, minibatch size, regularization type, and coefficient.Source: §3.2.9 · p. 37
• For manual tuning, validation data are extracted from the training set, and the setting with minimum validation error is selected before test data estimate expected risk.As hypothesis-space capacity grows, training error typically decreases while validation error first decreases and then rises; larger capacity enters the overfitting regime.Source: §3.2.9 · pp. 37–38, Figure 15 · p. 38
• When datasets are small, k-fold cross-validation reduces reliance on fixed subsets by averaging validation errors across disjoint training-validation splits for each hyperparameter setting.The hyperparameter instance with the smallest averaged validation error is selected, and the chosen model is then trained on the original training set.Source: §3.2.9 · p. 38
• Grid search tests hyperparameter combinations exhaustively, random search samples configurations, and Bayesian optimization adapts its search based on prior results.Bayesian optimization is described as often leading to faster convergence to optimal hyperparameters.Source: §3.2.9 · p. 38

3.3.1 Model components

• Fully connected layers apply affine transformations without respecting signal structure, while convolution and pooling extract local grid features and reduce signal size.Convolution applies a trainable kernel repeatedly to local regions; pooling can summarize values by a maximum or average.Source: §3.3.1 · p. 39
• Dropout randomly removes neurons during training to reduce overfitting, while normalizing layers facilitate training by setting output mean and variance to 0 and 1.Dropout can be interpreted as noise injection; its hyperparameter p specifies the percentage of neurons turned off.Source: §3.3.1 · pp. 39–40
• Attention layers compare token queries and keys to weight values, while positional encodings supply token order and multiple heads elaborate distinct input representations.Tokens are embedded before positional information is added; multi-head outputs are concatenated and projected.Source: §3.3.1 · pp. 40–41

3.3.2 Architectures

• Architecture categories are flexible rather than rigid: networks and layers can cross learning paradigms, while feed-forward networks serve classification, regression, and pattern recognition and CNNs process grid inputs.CNNs extract features from local spatial patterns and are applied to image classification, object detection, and facial recognition.Source: §3.3.2 · pp. 41–42
• RNNs process sequences through hidden states that retain history, whereas autoregressive models explicitly pass memory through prior outputs, often over a fixed-length window.Traditional RNNs face exploding and vanishing gradients, and their sequential structure precludes parallelization in training; gated RNNs, LSTMs, and GRUs bypass the gradient drawbacks.Source: §3.3.2 · pp. 42–43
• GNNs represent graph entities with node and edge labels and update node states from neighboring nodes and edges; encoder-decoder networks map an input sequence through a code to an output sequence.GNN fixed-point updates have a unique solution when the transition function is a contraction.Source: §3.3.2 · pp. 43–44
• Transformers use multi-head attention to exploit parallelism and achieve efficiency and scalability, and attention models outperform sequential RNNs.The overview identifies NLP, machine translation, text generation, and image classification as attention-model applications.Source: §3.3.2 · pp. 44–45
• Transformers combine attention and feed-forward layers in encoder–decoder structures, while Vision Transformers turn image patches into tokens for image tasks.The transformer encoder uses attention and feed-forward sub-layers with residual connections and normalization; its decoder generates output tokens iteratively. ViT projects image-patch tokens and adds positional encodings.Source: §3.3.2 · pp. 45–46
• Swin Transformers use alternating regular and shifted local windows with hierarchical image representations; SwinV2 changes normalization and attention to address training instability in deep networks and large images.SwinV2 also uses log-spaced relative-position bias and SimMIM self-supervised pretraining to reduce the need for large labeled image sets.Source: §3.3.2 · pp. 47–49
• Unsupervised and decision-making architectures include adversarial generator–discriminator networks, input-reconstructing autoencoders, environment-interacting reinforcement learning, and hybrids combining network types.The section names applications including image generation and augmentation, dimensionality reduction, game playing, robotics, and vision tasks.Source: §3.3.2 · pp. 49–50
• Deep architectures must balance training ease, prediction accuracy, memory footprint, computational cost, and scalability.The section illustrates model scale with examples including GPT-3 at 96 layers and 175 billion parameters, and GPT-4 at 120 layers and 1500 billion parameters.Source: §3.3.2 · p. 50

3.3.3 Ultimate generation hardware: GPUs and TPUs

• GPUs with tensor cores and TPUs accelerate neural-network tensor operations, but hardware efficiency also depends on minimizing GPU–CPU data transfers and their latency, bandwidth, synchronization, memory-coherency, and power costs.Multiple GPUs can scale processing power, and dedicated tensor cores and TPUs are presented as steps toward energy-efficient training and inference.Source: §3.3.3 · pp. 50, 52

3.4 Topics related to Machine Learning not covered in this paper

• The paper limits its coverage to selected topics because of space and encourages readers to explore other relevant ML topics through the cited references.Source: §3.4 · p. 52
• Proper initialization is crucial for effective neural-network training; Xavier and He methods help mitigate vanishing or exploding gradients.Such gradients can hinder effective learning.Source: §3.4 · p. 52
• Transfer learning transfers knowledge between related tasks and can reduce training time and improve performance; federated learning trains across devices holding local data without exchanging it.Source: §3.4 · p. 52
• Explainable AI develops methods to make ML outputs understandable to humans, enhancing trust and transparency.Source: §3.4 · p. 52
• Neural Tangent Kernel theory helps explain training behavior in infinitely wide neural networks.Source: §3.4 · p. 52
• Overparameterization and double descent describe cases where adding parameters beyond a certain point can improve performance, contrary to traditional bias–variance expectations.Source: §3.4 · p. 52

• SciML is an interdisciplinary field that combines physics-based computational models with machine-learning algorithms for scientific and engineering applications.Source: §4 · p. 53
• Machine learning can recover unknown constitutive laws, calibrate parameters, assess sensitivity, reduce models, and accelerate digital-model solvers.Source: §4 · p. 53
• Digital models can add physical structure to machine learning by regularizing losses, augmenting scarce training data, and supporting operator learning.Source: §4 · p. 53
• Data-driven models may fit training data yet generalize poorly or produce physically unacceptable predictions without scientific oversight.The section also states that domain knowledge can reduce data needs and improve robustness, generalization, accuracy, interpretability, and defensibility.Source: §4 · p. 54
• SciML challenges include embedding problem structure in ML, improving digital-model description and efficiency, and handling noisy, sparse, irregular real-world data.Source: §4 · pp. 54–55
• The appropriate balance of physics and data ranges from digital models alone, through combined models, to data-driven methods when physics is largely unknown.Source: §4 · p. 55
• The section distinguishes physics-based and data-driven approaches to combining models, while cautioning that this classification is not rigid.Physics-based workflows can use ML for missing data, constitutive laws, operator learning, latent dynamics, or reduced-order models; data-driven workflows can use digital models to enrich training data or replace algorithm components.Source: §4 · pp. 55–56
• Observational, inductive, and learning biases provide distinct routes for introducing physical information into machine learning.They respectively use high-fidelity model data, choices such as hypothesis spaces or specialized architectures, and physical penalties in the loss.Source: §4 · p. 56

• A surrogate model is trained on input-output pairs generated by a full-order model to approximate its high-fidelity solution at lower computational cost.Source: §4.1 · p. 57
• nquery > N is the stated condition under which surrogate-model use is advantageous in the cost comparison.Training is usually expensive, so the approach is not competitive with a high-fidelity model for a single direct PDE solve; many-query uses include sensitivity analysis, parameter estimation, and uncertainty quantification.Source: §4.1 · p. 57
• For real-time applications, surrogate training can be performed offline and its cheaper solution evaluated online.Source: §4.1 · pp. 57–58
• A surrogate of a black-box solver can provide a differentiable approximation of its input-output behavior for optimization and sensitivity analysis.Source: §4.1 · p. 58
• The pointwise neural-network approach is resolution-invariant and meshless, making it flexible across training resolutions and typically lighter than the full-field alternative.Its lower-dimensional inputs and outputs yield fewer trainable parameters and often better generalization for the same training-data amount.Source: §4.1 · p. 59, Figure 31 · p. 59

4.2 Physics–Informed learning

• Physics-informed learning adds governing equations to an ML loss and can be adapted to forward PDE solutions, inverse problems, parameter estimation, experimental fitting, or noisy-data resolution.PINNs can approximate integer-order, integro-differential, fractional, and stochastic PDEs, including cases with missing data.Source: §4.2 · p. 59, §4.2.1 · p. 61
• Neural networks can serve as mesh-free trial spaces but not Galerkin test spaces; PDE-learning alternatives include strong-form PINNs [57], weak-form VPINNs [130] [131], and energy-based Deep Ritz.Strong-form solutions may not exist when PDE coefficients are discontinuous or pointwise loads are present.Source: §4.2 · pp. 60–61

4.2.1 Physics–Informed Neural Networks (PINNs)

• For forward solutions, PINNs minimize PDE and boundary residuals at interior and boundary collocation points, using automatic differentiation and gradient-based optimization.The collocation-point sums approximate integrals; random points correspond to Monte Carlo quadrature, while carefully selected weighted points may improve accuracy.Source: §4.2.1 · pp. 61–62
• PINNs are mesh-free and apply to irregular domains, but their optimization cost is typically higher than FEM’s, so traditional methods usually outperform them on well-posed forward problems; high-dimensional problems are an alternative use case.Mesh construction becomes challenging in high dimensions because of the curse of dimensionality.Source: §4.2.1 · pp. 62–63
• Fully connected PINNs struggle with high-frequency and multiscale solutions because of spectral bias, while periodic activations and adaptive or weighted losses can mitigate it.The cited approaches aim to improve representation across spectral scales and support solving problems with wave-like behavior or sharp discontinuities.Source: §4.2.1 · p. 63
• Under a stability assumption, the PINN error estimate shows that minimizing PDE and boundary residuals controls generalization error when enough collocation nodes are used.The estimate also includes a quadrature-error term, whose order depends on the quadrature formula’s accuracy.Source: §4.2.1 · p. 64
• For inverse problems, PINNs require only a minimal modification to the direct setup and can still work when data, such as boundary conditions, are missing.The illustrated parameter-identification setup estimates a PDE coefficient from observations of the solution.Source: §4.2.1 · pp. 64–65
• Multifidelity PINNs address data-scarce, ill-posed inverse problems by combining larger low-fidelity datasets with high-fidelity data; the high-fidelity model adds a nonlinear correction to a linear transform of the low-fidelity model.This structure is intended to improve optimization convergence and robustness to noise.Source: §4.2.1 · p. 65
• For parameter-dependent solutions, PINNs can use a solver-trained low-fidelity surrogate as a prior for a high-fidelity model, balancing computational efficiency with accuracy.The multi-fidelity formulation explicitly represents parametric dependence, making it advantageous when parameter variations significantly affect the solution.Source: §4.2.1 · p. 66
• PINNs can address ill-posed and small-data problems, even when initial or boundary conditions are missing or incompletely known.In data fitting, physics terms act as regularizers favoring solutions consistent with known physical principles.Source: §4.2.1 · p. 67
• PINNs impose Dirichlet conditions either through a boundary-residual penalty or through a lifting-and-mask solution form that satisfies the boundary values.The passage also notes constrained optimization via the augmented Lagrangian method as another strategy.Source: §4.2.1 · p. 67

4.2.2 Variational PINNs (VPINNs)

• VPINNs minimize a Petrov–Galerkin residual; their weak formulation lowers derivative order and can yield analytic losses and require fewer quadrature nodes than strong-form PINNs.These properties can make algorithms less computationally expensive and estimates more accurate in particular cases.Source: §4.2.2 · p. 68

4.2.3 Deep Ritz Method (DRM)

• The Deep Ritz Method minimizes an energy functional but applies only to PDEs with energy formulations; its loss can be non-convex and have local minima.Compared with PINNs, DRM avoids residual-weight tuning, but Dirichlet conditions and extensions to defective or inverse problems are less easy.Source: §4.2.3 · p. 68

4.2.4 Optimizing a DIscrete Loss (ODIL)

• ODIL discretizes PDEs on a grid and minimizes discrete residuals, preserving the accuracy and conservation properties of the employed full-order discretization.It can solve forward and inverse problems, including field reconstruction and body-shape inference from noisy data.Source: §4.2.4 · pp. 68–69
• Across three forward tests and one inverse test, ODIL outperforms PINNs in accuracy and computational costs.The tests cover the 1D wave equation, 2D Poisson equation, 2D lid-driven cavity, and conductivity inference from temperature.Source: §4.2.4 · p. 70

4.3 Operator learning

• Operator-learning surrogates use high-fidelity simulation data to provide rapid approximations in many-query settings, while avoiding repeated retraining for different boundary conditions or discretizations.Conventional neural-network architectures depend strongly on the discretization and boundary conditions used to generate training data.Source: §4.3 · pp. 70–71

4.3.1 Deep Operator Networks (DeepONet)

• DeepONet combines a branch net that processes input-function values at sensors with a trunk net that predicts shape functions at output points.The theorem guarantees approximation of continuous operators under its assumptions, but a sufficiently rich network may require many sensors, and approximation error alone does not ensure optimization or generalization efficacy; deeper subnetworks are proposed to reduce sensor requirements and computational costs.Source: §4.3.1 · p. 72

4.3.2 Neural Operators

• Neural Operators generalize vector-to-vector networks to function-to-function mappings, with discretization invariance and universal approximation as key properties.Their kernel layers act on functions over continuous domains; discretization invariance lets fixed parameters operate across input discretizations, evaluate outputs at any point, and converge toward an infinite-dimensional operator as the discretization is refined.Source: §4.3.2 · pp. 72–73
• FNO parameterizes its kernel in frequency space and is discretization invariant, but its parameters can grow rapidly unless output Fourier coefficients decay quickly.FNO is more capable than convolutional neural networks at learning low frequencies, with larger errors at higher frequencies; a second FNO can be trained to predict the first model’s residual to improve high-frequency capture.Source: §4.3.2 · p. 74

4.3.3 Operator learning for time–dependent problems

• For time-dependent problems, Neural ODEs learn an operator mapping initial conditions and input trajectories to output trajectories by approximating the system’s temporal derivatives with a neural network.Derivative-based training fits derivatives estimated from data, whereas trajectory-based end-to-end training fits numerically integrated trajectories; the latter is more computationally challenging, and the adjoint method can reduce gradient-memory requirements.Source: §4.3.3 · pp. 75–76
• Trajectory-based Neural ODE training is typically more robust to noisy data and long-term predictions than derivative-based training, whose errors can accumulate during integration.Autoregressive models can extrapolate beyond sampled times, but reliable predictions depend on representative training data covering the relevant dynamics.Source: §4.3.3 · pp. 76–77
• SINDy identifies sparse ODEs as interpretable combinations of candidate functions, and is particularly effective when only a few terms govern the true dynamics.Candidate-library size and type affect expressive capacity and overfitting risk; sparse regression promotes parsimonious models [40, 39].Source: §4.3.3 · pp. 77–78

4.3.4 Intrinsic or hidden dynamics discovery

• For high-dimensional states, low-dimensional representations can improve interpretability and computational efficiency, but projection-based ROMs struggle when dynamics resist linear compression.Nonlinear autoencoders offer an alternative, while nonlinear-term projection can require costly hyper-reduction that trades accuracy against computational savings.Source: §4.3.4 · pp. 78–79
• Learning latent variables jointly with dynamics can yield representations suited to both reconstruction and prediction, with fewer parameters than end-to-end autoencoders and flexibility for nonuniform spatial sampling.Unlike a pre-trained autoencoder’s fixed latent space, model learning discovers the latent representation alongside the dynamics.Source: §4.3.4 · pp. 80–81
• Latent-variable model learning can discover hidden dynamics without modification, capturing unobserved internal information and non-Markovian effects.Unlike autoencoder-based methods, it does not require an encoder; delay-embedding alternatives require regularly sampled observations.Source: §4.3.4 · p. 81

4.3.5 Space-time operator learning

• Space-time operator learning maps system inputs and state to spatio-temporal outputs; treating time distinctly can preserve temporal direction and support time extrapolation.DeepONets and Neural Operators can instead treat time as another spatial variable, but time-aware methods may offer better generalization.Source: §4.3.5 · pp. 81–82
• A common space–time strategy semi-discretizes spatial fields into high-dimensional vectors and learns temporal dynamics, simplifying the problem but potentially losing spatial structure and retaining dimensionality challenges.Source: §4.3.5 · p. 82
• LDNets represent fields continuously as coordinate-conditioned neural networks and learn latent variables jointly with dynamics, enabling resolution-independent spatial representations and potentially non-Markovian dynamics.Source: §4.3.5 · p. 83
• LDNets are meshless and resolution-invariant, accept variable sampling, support query-based stochastic training, and in tested cases outperform autoencoder methods with orders of magnitude fewer trainable parameters.Source: §4.3.5 · p. 83

4.3.6 Foundation models for operator learning

• Poseidon combines a lead-time-conditioned SwinV2 U-Net scOT backbone with all2all semigroup-based trajectory training and generalizes from PDE pretraining to unseen, unrelated physics.Source: §4.3.6 · pp. 84–85
• ICON uses examples and a question condition as a prompt to predict a requested function value without fine-tuning, while supporting variable numbers of examples and key–value sequence lengths [259].Source: §4.3.6 · p. 86
• ICON generalizes to finer or smaller meshes, operators outside the training distribution, and new operator forms not seen during training [259].Source: §4.3.6 · pp. 86–87

4.4 Topics related to Scientific Machine Learning not covered in this paper

• The topics are selected for thematic coherence, not to label some areas of SciML more important than others.The authors describe the field’s possible methodological and application-specific combinations as virtually limitless.Source: §4.4 · p. 87
• Emerging directions include discovering constitutive laws, learning PDE solution operators across variable domains, and using machine learning to accelerate scientific computing.Examples of acceleration include learning artificial viscosity models or optimal stabilization parameters.Source: §4.4 · p. 87
• Other emerging topics include SciML for linear algebra and domain decomposition, learning corrections to physical models, and links between machine learning and control theory.Source: §4.4 · p. 87

5.1 The integrated heart model

• The iHeart simulator integrates cardiac electrophysiology, mechanics, blood and valve dynamics, circulation, perfusion, and ECG generation to model heart function and support patient-specific calibration.Its simulations use the lifex high-performance library for multiphysics and multiscale problems [2] [3] [5] [42].Source: §5 · p. 87, §5.1 · pp. 90, 92
• Myocardial deformation feeds back into electrophysiology through deformation-dependent conductivity and couples to blood dynamics through fluid–structure interaction.Blood and valve dynamics are modeled with incompressible Navier–Stokes equations in a moving domain using the ALE framework and the RIIS method for valves.Source: §5.1 · p. 90

5.2 Multifidelity PINNs for the estimation of ionic parameters

• Multifidelity PINNs are applied to estimate the BO ionic model parameter τfi from time-dependent transmembrane potential and gating-variable measurements.The fully connected network takes time as input and has four outputs for u, v, w, and s; the study also applies a standard PINN.Source: §5.2 · pp. 93–94
• The BO-model training data are generated with a finite-difference solver using a 0.1 ms time step through 0.8 s, with potential sampled every 25 ms and gating variables every 0.3 s.Zero-mean Gaussian noise with standard deviation σ is added to the transmembrane-potential data.Source: §5.2 · p. 93
• MPINN achieved significantly lower ionic-parameter estimation errors than PINN at both noise levels and accelerated convergence by using the low-fidelity model as a prior.The study presents multifidelity PINNs as a robust option for highly nonlinear estimation with noisy, sparse data.Source: §5.2 · pp. 94–95, Table 3 · p. 95

5.3 Physics–aware NNs for the inverse problem of electrocardiography

• ST–RB–DNN reconstructs epicardial potentials and torso signals from body-surface data; its autoencoder training gave slightly lower 12-lead ECG errors, while the alternative was more accurate on epicardial activation maps.DFT preprocessing improved performance for models trained on 12-lead ECG signals; the conventional inverse problem also involves a difficult-to-tune regularization parameter and conductivity inputs inferred from radiation-involving CT scans.Source: §5.3 · pp. 95, 97

5.4 Learning the microscopic dynamics in the framework of a coupled multiscale problem

• Adding cycle and equilibrium conditions made the microscale surrogate more stable and accurate than data fitting alone, with the best results from strongly imposing equilibrium.The cycle condition encourages the latent state to return to its initial condition when the full-order trajectory returns to the tele-diastolic state.Source: §5.4 · pp. 98–99, Table 4 · p. 99
• Less than 2% average error and over 1000 times faster resolution were reported for a two-variable surrogate that generalized to tests beyond training.When deployed at each macroscale node, scale interactions damped surrogate errors; coupled biomarker errors were on the order of 10^-3, while per-node variables fell from 2198 to 24.Source: §5.4 · p. 99, Table 5 · p. 100, Figure 45 · p. 100

5.5 Time-dependent operator learning for a multiphysics coupled problem

• The multiphysics operator-learning use case targets faster evaluation of the coupled cardiac electromechanics and blood-circulation input-output map for exploring model parameters and clinically relevant outputs.The systems interact strongly: muscle contraction affects circulatory flows and pressures, which in turn influence cardiac deformation.Source: §5.5 · p. 101
• The coupled surrogate reproduced full-order model results accurately in 10-heartbeat simulations despite training on five-heartbeat transients, with no significant difference between one- and four-parameter cases.The training simulations did not reach a limit cycle, yet the surrogate remained accurate when extrapolating to longer simulations.Source: Table 6 · p. 103, §5.5 · p. 103
• Time-dependent operator learning surrogates only the costly 3D electromechanics model while retaining the inexpensive 0D circulation model, reducing training-simulation needs and capturing dynamics that static emulators miss.The approach [210] was reported to predict accurately in long simulations, whereas static emulators often require 1,000 or more training samples and fail to capture temporal dynamics.Source: §5.5 · pp. 101, 103
• The ROM reduced per-heartbeat computation from approximately 4 hours on 32 cores for the FOM to less than a second on one laptop core, a speed-up over 10^5.Training-data generation and training costs must also be considered; the ROM is especially advantageous for repeated parameter exploration such as sensitivity studies and parameter estimation.Source: §5.5 · p. 103

5.6 NN–based surrogate models for global sensitivity analysis

• 3,300 times: surrogate-based global sensitivity analysis took 7.5 days rather than around 68 years for the FOM on a 160-core cluster.The surrogate workflow included 6.7 days for dataset generation, 18 hours for training, and 1 hour and 17 minutes for QoI computation; the scenario required 74,000 parameter sets and 740,000 heartbeats.Source: §5.6 · pp. 104–105

5.7 NN–based surrogate models for Bayesian parameter estimation

• Accounting for surrogate approximation error alongside experimental measurement error gives total covariance Σ = ΣROM + Σexp in Bayesian parameter estimation.ΣROM can be estimated from a validation set, while Σexp is determined by the measurement protocol.Source: §5.7 · p. 105
• At each tested noise level, the 90% posterior credibility region included the true parameter values; higher noise widened the regions, and their oblique shape reflected correlated parameter estimates.The measured pressures could be similarly affected by changes in contractility and arterial resistance.Source: §5.7 · p. 106, Figure 46 · p. 106
• 5,000x speed-up: Bayesian estimation requiring approximately 960,000 heartbeats took 6 days and 8 hours with the surrogate, versus over 87 years with the FOM.For a new dataset, retraining is unnecessary; rerunning MCMC takes 13 hours and 20 minutes.Source: §5.7 · pp. 106–107

5.8 Latent Dynamics Networks to accelerate electrophysiology simulations

• Cardiac electrophysiology simulations help explain and predict tissue electrical behavior and can support arrhythmia diagnosis, treatment planning, device design, and emergency decision-making.The passage frames these as clinical applications of such simulations.Source: §5.8 · p. 107
• Steep transmembrane-potential gradients, nonlinearities, multiscale coupling, depolarization fronts, and wave collisions make electrophysiology simulations costly and challenging for space-time operator learning.The test cases used a 1D domain with two stimulation points and a 2D domain with a propagating wave followed by a circular stimulus.Source: §5.8 · p. 107
• The comparison tested POD/DEIM, autoencoder models with Neural ODE or LSTM dynamics, end-to-end optimized variants, and LDNets on shared solver-generated datasets.The models predicted transmembrane potential from applied stimuli in 1D and 2D test cases.Source: §5.8 · pp. 107–108
• POD-DEIM reconstruction accuracy was reduced with 12 modes; satisfactory results required more than 24 modes, increasing prediction cost, while its speed-up remained minimal [209].POD-DEIM must use the high-fidelity model’s temporal discretization for numerical stability, constraining prediction speed.Source: §5.8 · p. 108
• LDNets achieved testing normalized RMSE of approximately 7×10−3 in both 1D and 2D, while autoencoder methods were approximately five times larger or more.LDNet predictions were nearly indistinguishable from the high-fidelity model and captured the three behaviors; autoencoder methods showed artifacts, especially when tissue refractoriness prevented propagation. Autoencoder methods required over ten times more parameters in 1D and over 400 times more in 2D.Source: §5.8 · p. 108

6 Some final thoughts and concluding remarks

• Tables 7 and 8 report training and test errors, trainable-parameter counts, and offline and online computational times for 1D and 2D test cases.Offline time covers model construction; online time covers predicting a new sample after construction.Source: Table 7 · p. 109, Table 8 · p. 109
• Least-squares regression is a learning process that minimizes mean square error on training data, and its model choice can be guided by physical knowledge.Such an assumption can be accurate in some circumstances and inappropriate in others; a linear stress–strain fit may fail to generalize when the true curve is exponential.Source: §6 · p. 110
• The model function is chosen a priori, but a rich neural-network hypothesis space can let the algorithm effectively discover a model from data.By contrast, selecting a linear or exponential stress–strain model imposes a prior assumption about the relationship.Source: §6 · p. 110
• Linear least squares uses an analytic gradient and solves normal equations, whereas nonlinear least squares typically iterates and performs well on small systems.Neural-network training typically uses backpropagation and stochastic gradient descent; after training, a neural network is deterministic for the same input, with exceptions noted for reinforcement learning, adversarial networks, and generative AI.Source: §6 · p. 110
• Neural networks can represent complex, high-dimensional input–output processes, but their many parameters and nonlinear compositional structure make them harder to interpret than simpler models.The paper gives ResNet-50 (25 million parameters) and AlexNet (60 million parameters) as examples.Source: §6 · p. 111
• Interpretability lies on a spectrum: simpler models offer clearer input–output relationships, while more complex models can capture intricate patterns at the cost of interpretability.For a linear stress–strain model, slope represents stiffness and intercept represents residual stress at zero strain.Source: §6 · p. 111
• SciML is relatively new, and its major achievements often rely on trial and error rather than an established theoretical framework; the paper’s classifications are preliminary.The authors expect these classifications to require revision as the field develops.Source: §6 · p. 111