Source-linked AI summary
Theory-guided hard constraint projection (HCP): a knowledge-based data-driven scientific machine learning method
Yuntian Chen, Dou Huang, Dongxiao Zhang, Junsheng Zeng, Nanzhe Wang, Haoran Zhang, Jinyue Yan
TL;DR
Scientific machine learning struggles to combine domain knowledge with incomplete, noisy observations while preserving physical consistency. The paper proposes theory-guided HCP, which discretizes governing equations and uses projection to impose hard constraints; experiments on heterogeneous subsurface flow report improved accuracy and robustness with sparse data, while outliers remain a limitation.
Problem
Scientific machine-learning models struggle to combine domain knowledge with incomplete, noisy observations and may produce physically inconsistent predictions.
Method
Theory-guided HCP discretizes governing equations and projects neural-network predictions within a constraint patch onto results satisfying the physical constraints.
Results
Experiments on heterogeneous subsurface flow show that theory-guided HCP performs better than soft-constraint and purely data-driven models, including under sparse observations and noise.
Takeaways & Limitations
Hard constraint projection enables neural networks to use sparse observations while enforcing governing equations in the constraint patch.
Takeaways & Limitations
Outliers degrade both theory-guided HCP and TgNN substantially, and TgNN becomes slightly more accurate than HCP when outliers exceed 4%.
Abstract
from arXiv · showhide
Machine learning models have been successfully used in many scientific and engineering fields. However, it remains difficult for a model to simultaneously utilize domain knowledge and experimental observation data. The application of knowledge-based symbolic AI represented by an expert system is limited by the expressive ability of the model, and data-driven connectionism AI represented by neural networks is prone to produce predictions that violate physical mechanisms. In order to fully integrate domain knowledge with observations, and make full use of the prior information and the strong fitting ability of neural networks, this study proposes theory-guided hard constraint projection (HCP). This model converts physical constraints, such as governing equations, into a form that is easy to handle through discretization, and then implements hard constraint optimization through projection. Based on rigorous mathematical proofs, theory-guided HCP can ensure that model predictions strictly conform to physical mechanisms in the constraint patch. The performance of the theory-guided HCP is verified by experiments based on the heterogeneous subsurface flow problem. Due to the application of hard constraints, compared with fully connected neural networks and soft constraint models, such as theory-guided neural networks and physics-informed neural networks, theory-guided HCP requires fewer data, and achieves higher prediction accuracy and stronger robustness to noisy observations.
1. Introduction
Scientific machine learning must combine rich domain knowledge with data-driven fitting, but soft constraints may not prevent local physical violations. The paper therefore motivates projection-based hard constraints for neural-network predictions.
- Data and domain knowledge: Data-driven models fit complex scenarios well but depend heavily on costly, incomplete, and noisy observations while lacking encoded physical mechanisms.This limits their ability to use accumulated domain knowledge during inference.
- Data and domain knowledge: Symbolic AI emphasizes domain knowledge, whereas connectionism AI emphasizes data, motivating models that combine both information sources.The paper frames this combination as a way to strengthen the information available for machine-learning inference.
- Soft constraints: Theory-guided and physics-informed models embed governing equations as loss-function regularizers, so they enforce physical consistency only in an average sense.These soft constraints cannot ensure that predictions satisfy the physical constraint at every point.
- Hard constraints: Hard constraints can strictly enforce governing equations in a specified region, potentially requiring fewer data and improving accuracy and robustness to noisy observations.The paper contrasts this optimization strategy with soft-constraint penalties.
- Projection-based approach: The proposed HCP discretizes governing equations, extracts a constraint patch, and projects neural-network predictions onto the constraint-satisfying set.The projected results strictly conform to the governing equation on the constraint patch and are trained end-to-end.
2. Methodology
Theory-guided HCP combines neural-network fitting with theory-guided losses and hard constraint projection, enforcing governing equations within local constraint patches while supporting sparse and partial observations.
- Theory-guided framework: The theory-guided framework combines observation, governing-equation, boundary, and initial losses through a linear combination of penalty terms.
- Hard constraint modeling process: Soft constraints may permit physical inconsistency, whereas hard constraints strictly satisfy a governing equation within a constraint patch.Soft constraints enforce conformity only in an average sense through regularization terms.
- Hard constraint modeling process: HCP constructs hard constraints through equation discretization, matrix decomposition, and projection, with the constraint patch representing local physical relationships.Discretization transforms governing equations into a discrete form that is easier to handle.
- Theory-guided framework: Theory-guided HCP combines a feedforward inference process with hard constraints and a backward process using backpropagation of theory-guided loss.The framework uses observations and domain knowledge, including boundary conditions, initial conditions, and governing equations.
- Sparse and partial observation: Theory-guided HCP requires only a small amount of labeled data and can use label-free collocation points when observations cover only part of the physical field.In the cited experiment, observations cover 18 of 50 time steps, leaving 32 time steps label-free.
- Sparse and partial observation: Theory-guided HCP can extrapolate beyond the training-data range, unlike conventional deep-learning methods that cannot guarantee extrapolation performance.The paper attributes this capability to introducing domain knowledge through hard constraints and the theory-guided framework.
- Intuitive understanding: The projection process is illustrated as moving predictions onto a physical constraint hyperplane before optimization toward observations, rather than optimizing freely off the constraint.The mountain analogy contrasts unconstrained gradient descent with movement along the constraint hyperplane.
3.1 Heterogeneous subsurface flow problem
The study evaluates theory-guided HCP on a two-dimensional transient saturated-flow problem with heterogeneous hydraulic conductivity, using early-time observations to predict later flow-field development. Experiments compare ANN, TgNN, and HCP under varying observations, collocation and boundary points, noise, and outlier proportions.
- Problem setup: The case study models transient saturated flow through a heterogeneous porous medium governed by a physical flow equation.The heterogeneous conductivity field is intended to better represent practical conditions.
- Problem setup: The simulation domain is a 51×51 grid representing a square with constant-pressure boundaries.The domain length is 1020 [L] in both directions.
- Data split: The dataset uses the first 18 of 50 hydraulic-head time steps for training observations and the final 32 for testing.The task predicts later flow-field development from sparse early-time observations.
- Evaluation: Direct ANN prediction is inaccurate for this extrapolation problem, while TgNN improves accuracy through soft constraints but requires many collocation and boundary points.These models provide the comparison context for evaluating HCP.
- Evaluation: Experiments compare ANN, TgNN, and theory-guided HCP across different numbers of collocation points, boundary points, and observations, plus noise and outlier conditions.Relative L2 loss is used to evaluate model performance.
3.2 Difference between hard constraints and soft constraints
The comparison examines how hard-constraint projection differs from soft-constraint training by tracking multiple losses and the size of projection adjustments. HCP’s loss decays faster than TgNN, while each projection changes fewer than 1% of prediction values in the illustrated patches.
- Loss comparison: Figure 8 compares comprehensive, observation, PDE, and condition losses for HCP and TgNN during the first 500 training epochs.The models use the same architecture and dataset.
- Loss comparison: HCP’s loss value decays faster than TgNN’s, indicating more effective practical embedding of physical constraints through projection.This comparison concerns the loss curves shown in Figure 8.
- Loss interpretation: The PDE loss can begin near zero because an all-zero flow field satisfies the governing equation, even though it violates boundary and initial conditions.The other losses therefore remain relevant during initial iterations.
- Projection effect: Less than 1% of prediction results are adjusted to convert prediction matrices into projected matrices satisfying physical constraints in the illustrated patches.The adjustment is small at each iteration but is associated with faster and more efficient optimization directions.
3.3 Data demand analysis
Theory-guided HCP generally maintains higher accuracy and robustness than TgNN and ANN when training data vary in quantity, especially with sparse collocation, boundary, or observation points. Its advantage is strongest in extrapolation settings where observations cover only earlier time steps.
- Collocation points: HCP achieves lower relative L2 loss than TgNN with fewer than 400 collocation points and shows more consistent performance across random experiments.At larger collocation-point counts, the two models become more similar in accuracy, but HCP remains more robust in the reported comparison.
- Collocation points: With only 40 collocation points, HCP predictions remain closer to ground truth than TgNN and ANN at the 30th and 50th time steps.These time steps lie beyond the first 18 observed steps, making the comparison an extrapolation test.
- Boundary points: HCP retains higher accuracy and robustness than TgNN as boundary points increase from 4 to 40,000 total points.The models gradually converge as boundary-point counts grow, but HCP remains superior even at 10,000 points per boundary.
- Observation points: HCP and TgNN perform significantly better than ANN in extrapolation, and HCP maintains lower residuals as observation counts increase.ANN accuracy first increases and then decreases, consistent with overfitting observations from the first 18 time steps.
3.4 Model robustness comparison
HCP and TgNN are comparatively robust to increasing observational noise, but HCP becomes vulnerable to outliers that contaminate a constraint patch. When outliers exceed 4%, TgNN slightly surpasses HCP, although HCP remains better than ANN.
- Noise: From 10% to 60% noise, HCP and TgNN performance does not deteriorate significantly, indicating limited sensitivity to noise.HCP has an obvious accuracy and robustness advantage when the noise level is below 1%, while the models gradually converge at higher noise levels.
- Noise: Domain knowledge helps HCP and TgNN distinguish noisy observations from physically consistent values, supporting their robustness relative to purely data-driven ANN.The stated mechanism is that the true values conform to the physical mechanism whereas ANN cannot distinguish true values from noise.
- Outliers: Both HCP and TgNN lose prediction accuracy significantly as the proportion of outliers increases.The experiment distinguishes outliers, which are gross individual errors, from noise, which is a general perturbation in observations.
- Outliers: When outliers exceed 4%, TgNN is slightly better than HCP, although HCP remains better than conventional ANN.An outlier inside a constraint patch can cause the projected values of other points in that patch to become incorrect.
4. Conclusion
The paper proposes theory-guided HCP to impose governing equations as hard constraints through projection. Experiments show that it can use sparse observations and achieve strong accuracy and robustness with fewer training data, while remaining sensitive to outliers.
- Contribution: Theory-guided HCP converts governing equations into hard constraints so predictions within a constraint patch strictly satisfy the differential governing equation.Unlike soft-constraint regularization, projection provides a mechanism for adjusting predictions onto a physically admissible set.
- Contribution: The three-step HCP procedure consists of equation discretization, matrix decomposition, and projection within the theory-guided framework.A projection matrix based on the constraint matrix maps prediction values onto a hyperplane satisfying the physical constraints.
- Contribution: HCP can predict subsequent flow-field time steps from sparse early-time observations under indirect supervision from domain knowledge.The method is designed to combine sparse labeled data with label-free collocation data and governing equations.
- Experimental findings: Experiments indicate that HCP reduces data demand and improves accuracy with fewer collocation, boundary, and observation points, while remaining robust to noise.The robustness analysis reports stable results under noisy observations.
- Limitation: HCP is sensitive to outliers because one corrupted value can distort the projected values throughout its constraint patch.The conclusion identifies this behavior as a limitation of the hard-constraint projection approach.
Appendix A: The proof of projection matrix
Appendix A derives projection formulas for least-squares fitting and for projecting predictions onto an affine constraint set. It also explains how the theory-guided HCP handles single constraint patches in mini-batch training.
- Projection onto a column space: When Aθ=Y* is overdetermined, minimizing the residual projects Y onto Range(A).The projection is the closest point Y* to Y in the column space of A.
- Projection onto a column space: Setting the residual gradient to zero yields θ*=(A^T A)^-1A^T Y when A^T A is invertible.This solution produces the projection Y*=A(A^T A)^-1A^T Y.
- Projection onto an affine constraint set: For m<n, the closest w* satisfying Aw=b is (I−A^T(AA^T)^-1A)w+A^T(AA^T)^-1b.The result follows by decomposing w into projections onto the affine constraint set and its orthogonal complement.
- Constraint regimes: Theorem 1 treats overdetermined constraints with m>n, whereas Theorem 2 treats a single constraint patch with m=1<n.The distinction determines whether the projection uses A^T A or AA^T.
- Mini-batch implementation: Mini-batch training represents the constraint matrix as A∈R^(b,m,n), keeping m as the number of constraints within each patch.Using the batch size as m would incorrectly treat the mini-batch as one constraint system.
Appendix B: Projection strategy based on ghost cell to process boundary conditions
Appendix B uses ghost cells to encode boundary conditions before projection. This modification lets projected predictions satisfy boundary conditions and the governing equation simultaneously.
- Ghost-cell construction: Ghost cells are virtual grid cells outside the boundary used to encode boundary-condition relationships.The approach is introduced for constant-pressure and no-flow boundaries.
- Constant-pressure boundary: For constant-pressure boundaries, ghost-cell hydraulic pressure and conductivity are adjusted so projected predictions retain the prescribed boundary pressure.The boundary cells, ghost cells, and neighboring cells must obey a specific relationship.
- No-flow boundary: For no-flow boundaries, setting ghost-cell hydraulic conductivity to 0 guarantees the boundary condition.The modified constraint and prediction matrices are then used before hard constraint projection.
Appendix C: Cost of theory-guided HCP and simulation to solve a physical field
Appendix C compares computational costs for simulation and the neural network as the time step increases. Simulation cost grows linearly, while neural-network cost remains basically unchanged.
- Computational-cost comparison: As the time step increases, simulation computational cost increases linearly.The simulation computes sequentially from the first time step to the tth time step.
- Computational-cost comparison: Neural-network computational cost remains basically unchanged as the time step increases.The network uses (t,x,y) as independent inputs rather than stepping through the simulation sequentially.
- Computational-cost comparison: The comparison measures simulation time on the red left axis and neural-network time on the blue right axis.
Appendix D: Experiment results of ANN, TgNN, and HCP
Appendix D presents relative L2 loss results for ANN, TgNN, and theory-guided HCP across different scenarios.
- Evaluation metric: Table D1 uses relative L2 loss as the reported evaluation metric.
- Compared methods: The table compares ANN, TgNN, and theory-guided HCP.
- Experimental scenarios: The results are organized across different scenarios.