Source-linked AI summary
MIM: A deep mixed residual method for solving high-order partial differential equations
Liyao Lyu, Zhen Zhang, Minxin Chen, Jingrun Chen
TL;DR
High-dimensional PDEs challenge classical numerical methods because their unknown counts grow exponentially with dimension. MIM rewrites high-order PDEs as first-order systems and fits their residuals in a least-squares loss; across the tested equations, it improves approximations over DGM, including for both derivatives and solutions.
Problem
Classical numerical methods designed for low-dimensional PDEs face high-dimensional problems where the number of unknowns grows exponentially with dimension.
Source: §1 · p. 2Method
MIM rewrites a high-order PDE as a first-order system and minimizes its equation residuals in the least-squares sense.
Source: §1 · p. 3Results
MIM provides better approximations than DGM across the tested PDEs, for both high-order derivatives and the PDE solution.
Source: §1 · pp. 3–4, §4 · p. 20Takeaways & Limitations
MIM offers exact boundary-condition enforcement and better high-order derivative approximations at almost the same cost, while also improving the PDE solution approximation.
Source: §4 · p. 20The reported ReLU success covers tested time-independent equations, not KdV, and the authors say deeper classical numerical analysis remains to be transferred.
Source: §4 · p. 21
Abstract
from arXiv · showhide
In recent years, a significant amount of attention has been paid to solve partial differential equations (PDEs) by deep learning. For example, deep Galerkin method (DGM) uses the PDE residual in the least-squares sense as the loss function and a deep neural network (DNN) to approximate the PDE solution. In this work, we propose a deep mixed residual method (MIM) to solve PDEs with high-order derivatives. In MIM, we first rewrite a high-order PDE into a first-order system, very much in the same spirit as local discontinuous Galerkin method and mixed finite element method in classical numerical methods for PDEs. We then use the residual of first-order system in the least-squares sense as the loss function, which is in close connection with least-squares finite element method. For aforementioned classical numerical methods, the choice of trail and test functions is important for stability and accuracy issues in many cases. MIM shares this property when DNNs are employed to approximate unknowns functions in the first-order system. In one case, we use nearly the same DNN to approximate all unknown functions and in the other case, we use totally different DNNs for different unknown functions. In most cases, MIM provides better approximations (not only for high-derivatives of the PDE solution but also for the PDE solution itself) than DGM with nearly the same DNN and the same execution time, sometimes by more than one order of magnitude. When different DNNs are used, in many cases, MIM provides even better approximations than MIM with only one DNN, sometimes by more than one order of magnitude. Therefore, we expect MIM to open up a possibly systematic way to understand and improve deep learning for solving PDEs from the perspective of classical numerical analysis.
1. Introduction
Classical numerical methods are well understood for low-dimensional PDEs but face the curse of dimensionality on high-dimensional problems. MIM recasts high-order PDEs as first-order systems and uses least-squares residuals, with choices in the unknown derivatives and network architecture; across four PDE types, it improves approximations over DGM.
1. Introduction
- Classical numerical methods are well understood for low-dimensional PDEs but face the curse of dimensionality on high-dimensional problems.The number of unknowns grows exponentially with dimension; deep-learning approaches have been developed for such PDEs.Source: §1 · pp. 2–3
- MIM rewrites a high-order PDE as a first-order system, approximating the solution and its high-order derivatives as unknown functions.This modeling choice is analogous to local discontinuous Galerkin and mixed finite element methods.Source: §1 · p. 3
- MIM permits choices of which derivatives are unknowns and whether one shared or multiple DNNs approximate them.The residuals of the resulting first-order system are combined in a least-squares loss, analogous to least-squares finite element methods.Source: §1 · p. 3
- MIM improves approximations over DGM across four PDE types, including both high-order derivatives and the PDE solution itself.The comparison includes Poisson, Monge-Ampère, biharmonic, and Korteweg-de Vries equations.Source: §1 · pp. 3–4
2. Deep mixed residual method
This section develops MIM’s loss construction and network setup: high-order derivatives are represented with auxiliary variables, while the choice of unknowns and network outputs depends on the formulation.
2.1. Loss function
- MIM represents high-order derivatives with auxiliary variables and allows different derivative subsets as unknowns; DGM’s loss measures PDE, initial-condition, and boundary-condition residuals.For biharmonic problems, the auxiliary variables include p = ∇u, q = ∆u, and w = ∇(∆u).Source: §2.1 · pp. 4–5
2.2. Neural network architecture
- For biharmonic problems, DGM predicts u, MIM1a predicts u and equation derivatives, and MIM2p uses separate networks for u and ∆u; MIM1 and DGM have close parameter counts.ResNet blocks combine two linear transforms and activations with a shortcut connection, a structure described as addressing vanishing or exploding gradients.Source: §2.2 · pp. 6–7, Figure 2 · p. 8
2.3. Stochastic Gradient Descent
- Training uses uniformly distributed points from the domain and its boundary, with ADAM optimization and automatic differentiation for derivatives in PyTorch.The learning rate is set to 1e-3.Source: §2.3 · p. 9
3. Numerical Result
Numerical experiments compare relative L2 errors across four PDEs. MIM generally outperforms DGM, while approximation quality also varies with the mixed formulation, activation function, and boundary treatment.
3.1. Poisson Equation
- For the Poisson Neumann problem, MIM uses multipliers on p to satisfy the exact boundary condition; how DGM imposes it exactly is unclear.The authors also note that MIM's access to both u and p allows boundary conditions to be imposed exactly in principle.Source: §3.1 · p. 10, Figure 3 · p. 11
- For Poisson, execution time per iteration ranks MIM1 < MIM2 < DGM, even though MIM2 has about twice DGM's parameters.The authors attribute MIM1's lower cost to automatic Neumann enforcement and similar network structures.Source: §3.1 · p. 10
- For Poisson, relative errors generally rank MIM2 < MIM1 < DGM for both u and ∇u; ∇u improves several times from DGM to MIM1 and about one order from MIM1 to MIM2.For u, the improvement is several times; Table 7 reports relative errors for u and ∇u.Source: §3.1 · p. 12, Table 7 · p. 12
- For Poisson at d = 4, MIM gives better u and ∇u approximations across tested activations and depths; ReLU suits MIM's first-order derivatives but not DGM's high-order derivatives.Table 8 fixes network width at 10.Source: §3.1 · p. 12, Table 8 · p. 14
3.2. Monge-Amp´ere equation
- For Monge–Ampère, results across network widths and dimensions show an obvious MIM advantage, with relative L2 errors evaluated using depth 2 and ReQU activation.Table 9 reports errors over widths and dimensions; Figure 6 plots the errors for u and ∇u.Source: §3.2 · p. 14, Table 9 · p. 15, Figure 6 · p. 16
3.3. Biharmonic equation
- For biharmonic problems, MIM improves u, ∇u, ∆u, and ∇(∆u) over DGM, while MIMp is slightly more accurate and comparable to MIMa despite fewer outputs.MIMa can enforce the exact boundary condition, although penalty terms are used in this example.Source: §3.3 · p. 16, Figure 7 · p. 17
3.4. KdV equation
- For KdV, MIM performs better than DGM, especially with ReQU activation, while no obvious improvement of MIM2 over MIM1 is observed.The reported relative L2 errors cover u, ∇u, and diag(∇2u).Source: §3.4 · p. 19
4. Conclusion and Discussion
MIM brings classical mixed and least-squares ideas to deep learning with flexible loss and network choices. The authors report better high-order derivative and PDE-solution approximations, while identifying deeper transfer of classical numerical analysis as future work.
4. Conclusion and Discussion
- MIM offers flexible loss-function choices and a larger solution space through flexible DNN choices.The method is motivated by local discontinuous Galerkin, mixed finite element, and least-squares finite element methods.Source: §4 · p. 19
- MIM provides better high-order derivative approximations at almost the same cost and also improves the PDE-solution approximation.The method is described as having potential to impose exact Neumann, mixed, and Robin boundary conditions, which DGM cannot enforce exactly.Source: §4 · p. 20
- The authors say only basic-level experience from classical numerical methods has transferred so far; deeper transfer and theoretical understanding remain future work.They report that ReLU works for tested time-independent equations but not for KdV.Source: §4 · p. 21