Source-linked AI summary
The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems
Weinan E, Bing Yu
TL;DR
Variational problems, especially those arising from PDEs, require numerical solution methods that can construct suitable functions in potentially high-dimensional settings. The paper proposes the Deep Ritz method, which uses deep neural networks within the Ritz framework and stochastic optimization, and reports accurate solutions, faster initial transfer-learning training, and potential advantages in dimensionality. The authors also identify unresolved optimization and convergence issues.
Problem
The paper addresses numerical solution of variational problems, particularly PDE-related problems, in settings where constructing functions in high dimensions is relevant.
Method
The Deep Ritz method represents trial functions with deep neural networks and combines numerical functional approximation with stochastic-gradient optimization.
Results
The method gives more accurate solutions than finite differences in a reported two-dimensional Poisson example, while transferred weights considerably speed initial training.
Takeaways & Limitations
The method has potential to work in rather high dimensions and is less sensitive to problem dimensionality, according to the authors’ limited experience.
Takeaways & Limitations
The resulting optimization problem is nonconvex, with local minima and saddle points, and there is no consistent conclusion about the convergence rate.
Abstract
from arXiv · showhide
We propose a deep learning based method, the Deep Ritz Method, for numerically solving variational problems, particularly the ones that arise from partial differential equations. The Deep Ritz method is naturally nonlinear, naturally adaptive and has the potential to work in rather high dimensions. The framework is quite simple and fits well with the stochastic gradient descent method used in deep learning. We illustrate the method on several problems including some eigenvalue problems.
1 Introduction
The paper proposes using deep neural-network function representations to solve variational problems, extending deep learning’s compositional approximation approach to PDE-related settings.
- Deep neural networks are proposed for constructing functions in contexts including partial differential equations and molecular modeling.
- The Deep Ritz method combines neural-network function representations with the classical Ritz method for solving variational problems.
- The method is introduced as a new algorithm whose features are explored later in the paper.
2 The Deep Ritz Method
The Deep Ritz method represents admissible trial functions with deep neural networks, approximates the variational functional numerically, and optimizes the resulting objective with stochastic gradient methods.
- The method has three components: neural-network approximation of trial functions, numerical quadrature of the functional, and optimization of the final problem.
- Building trial functions: A deep network maps x to zθ(x) and uses blocks containing two linear transformations, two activations, and a residual connection.
- Building trial functions: The residual connection is intended to ease training by helping avoid the vanishing-gradient problem.
- Building trial functions: The network output is inserted into the variational functional, producing an objective expressed in the network parameters for minimization.
- The stochastic gradient descent algorithm and the quadrature rule: The integral is approximated because explicit computation for neural-network trial functions is impractical.
- The stochastic gradient descent algorithm and the quadrature rule: SGD evaluates a randomly selected term or mini-batch instead of the full sum, reducing each iteration’s function evaluations relative to gradient descent.
- The stochastic gradient descent algorithm and the quadrature rule: For the variational integral, randomly sampled points in the domain serve as data points, with a common quadrature weight used at each point.
- The stochastic gradient descent algorithm and the quadrature rule: Random sampling avoids the fixed-node risk that the integrand is minimized at quadrature nodes while the full functional remains poorly minimized.
3 Numerical Results
The Deep Ritz method is evaluated on Poisson, high-dimensional, Neumann-boundary, weight-transfer, and eigenvalue problems, with results reported across dimensions and against finite differences. The experiments show accurate solutions, high-dimensional applicability, faster initial training with weight transfer, and deteriorating eigenvalue results as dimension increases.
- 3.1 The Poisson equation in two dimension: The Poisson corner-singularity test reports maximum error for both the Deep Ritz and finite difference methods.The error is defined as e = max |u*(x) − uh(x)| and is reported in Table 1.
- 3.1 The Poisson equation in two dimension: The Deep Ritz method gives more accurate solutions than the finite difference method for the Poisson problem.The comparison uses a penalty method for the boundary condition and finite differences on a uniform grid.
- 3.2 Poisson equation in high dimension: After 50,000 iterations, the relative error reaches about 2.2% for a problem with d = 100.The experiment uses a similar network structure to the preceding high-dimensional test.
- 3.4 Weight transfer: Transferring weights speeds up training considerably during its initial stage when the forcing function changes.The authors suggest weight transfer is particularly effective when the accuracy requirement is not very stringent.
4 Discussion
The Deep Ritz method represents trial functions with deep neural networks and offers adaptive, high-dimensional, and SGD-compatible variational optimization. The authors also identify unresolved optimization, convergence, boundary-condition, and design issues.
- The method represents trial functions by deep neural networks and is presented as a variational method.
- Its natural adaptivity and reduced sensitivity to dimensionality suggest potential for problems in rather high dimensions.
- The framework is reasonably simple and fits the stochastic gradient descent framework commonly used in deep learning.
- The resulting variational problem is nonconvex even when the initial problem is convex, making local minima and saddle points non-trivial issues.
- There is currently no consistent conclusion about the convergence rate, and essential boundary conditions are harder to treat than with traditional methods.
- Network structure, activation functions, and minimization algorithms remain open design issues for future work.