Source-linked AI summary
DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
Lu Lu, Pengzhan Jin, George Em Karniadakis
TL;DR
The paper asks how to learn nonlinear operators effectively when universal approximation alone does not address optimization and generalization errors. It proposes DeepONets with separate branch and trunk subnetworks, and reports improved generalization, sensor-dependent error bounds, and high-order convergence across dynamic-system and PDE problems.
Problem
Universal operator approximation does not determine how to learn operators effectively while controlling optimization and generalization errors.
Method
DeepONet encodes input-function values at fixed sensors with a branch network and output locations with a trunk network, then merges their representations.
Results
Across four ODE and PDE problems, DeepONets achieve small generalization errors and exhibit polynomial and even exponential error convergence in the simulations.
Takeaways & Limitations
The results support using a two-subnetwork inductive bias to learn nonlinear operators under consistent input sensors and unconstrained output locations.
Takeaways & Limitations
The paper lacks theoretical explanations for DeepONet generalization errors and network-size bounds for operator approximation.
Abstract
from arXiv · showhide
While it is widely known that neural networks are universal approximators of continuous functions, a less known and perhaps more powerful result is that a neural network with a single hidden layer can approximate accurately any nonlinear continuous operator. This universal approximation theorem is suggestive of the potential application of neural networks in learning nonlinear operators from data. However, the theorem guarantees only a small approximation error for a sufficient large network, and does not consider the important optimization and generalization errors. To realize this theorem in practice, we propose deep operator networks (DeepONets) to learn operators accurately and efficiently from a relatively small dataset. A DeepONet consists of two sub-networks, one for encoding the input function at a fixed number of sensors $x_i, i=1,\dots,m$ (branch net), and another for encoding the locations for the output functions (trunk net). We perform systematic simulations for identifying two types of operators, i.e., dynamic systems and partial differential equations, and demonstrate that DeepONet significantly reduces the generalization error compared to the fully-connected networks. We also derive theoretically the dependence of the approximation error in terms of the number of sensors (where the input function is defined) as well as the input function type, and we verify the theorem with computational results. More importantly, we observe high-order error convergence in our computational tests, namely polynomial rates (from half order to fourth order) and even exponential convergence with respect to the training dataset size.
1 Introduction
DeepONet addresses the gap between universal operator approximation and effective learning by separating input-function and output-location representations. The paper motivates this architecture for dynamic systems and PDEs under weak sensor constraints.
- Motivation: Single-hidden-layer neural networks can approximate nonlinear continuous operators, but the theorem does not explain effective learning from data.The practical error also depends on optimization and generalization, motivating an architecture beyond approximation capacity alone.
- Problem setup: DeepONet uses input-function values at fixed sensors and output locations as separate network inputs.The operator network receives [u(x1), u(x2), ..., u(xm)] and y, producing G(u)(y).
- Problem setup: The training data require identical sensor locations for input functions but impose no constraints on output-function evaluation locations.Sensors may be scattered rather than arranged on a lattice.
- Contribution: The proposed architecture uses a branch net for the input function and a trunk net for the output-function location to reduce total error.The paper specifically claims improved generalization from this two-subnetwork design.
- Applications: The study considers operator identification for dynamic systems and partial differential equations.These applications include nonlinear dynamic-system identification and PDE-related mappings.
2 Methodology
The methodology represents operators through separate branch and trunk subnetworks, with a more efficient unstacked variant and fully connected subnetworks used in the study. Training functions are sampled from Gaussian random fields and Chebyshev-polynomial spaces, then paired with numerical reference solutions.
- Network design: The baseline concatenates sampled input-function values and the output location, whereas DeepONet processes them through separate subnetworks.The FNN baseline receives [u(x1), ..., u(xm), y] as one input vector.
- Network design: The trunk network maps y to p outputs, while branch networks map sampled input-function values to coefficients that are merged with trunk outputs.This structure reflects the distinct roles of the input function and evaluation location.
- Network variants: The stacked architecture uses p parallel branch networks, while the unstacked architecture uses one branch network producing a p-dimensional vector.The unstacked form reduces the computational and memory cost of maintaining many branch networks.
- Implementation: The study uses fully connected networks for both DeepONet subnetworks to isolate the effect of the DeepONet architecture.The authors note that convolutional layers could potentially improve accuracy but generally suit equispaced square-grid sensors.
- Data generation: Input functions are drawn from Gaussian random fields and Chebyshev-polynomial spaces with controlled smoothness or basis complexity.For the GRF, the length scale l controls smoothness, with larger l producing smoother functions.
- Data generation: Reference outputs are generated by solving ODEs with Runge-Kutta (4, 5) and PDEs with a second-order finite-difference method.Each data point is a triplet (u, y, G(u)(y)), so one input function can yield multiple points at different locations.
3 Number of sensors for identifying nonlinear dynamic systems
The sensor analysis bounds operator-identification error through interpolation of the input signal and shows that the required sensor count depends on the input-function space. For Gaussian random fields with an RBF kernel, the interpolation bound scales inversely with m^2l^2.
- Operator setting: The dynamic-system operator maps an input signal u to the solution s of an ODE system.The output signal is the solution s:[a,b]→R^K, and G maps u to s.
- Sensor approximation: The input function is approximated from m+1 uniformly spaced sensor points using a piecewise-linear interpolant.The construction uses x_j=a+j(b−a)/m and defines u_m between neighboring sensors.
- Theoretical guarantee: Under Lipschitz assumptions on the ODE right-hand side, a sufficiently large sensor count yields an FNN approximation error below ε.The theorem requires c(b−a)κ(m,V)e^{c(b−a)}<ε and guarantees the stated bound for all u∈V.
- Approximation bound: The interpolation error satisfies max_x∈[a,b]|u(x)−u_m(x)|≤κ(m,V), with κ(m,V)→0 as m→∞.The bound is uniform over the compact input-function space V.
- Approximation bound: For a Gaussian random field with an RBF kernel, κ(m,V) scales as 1/(m^2l^2).Thus the sensor-related approximation bound depends on both sensor count and the kernel length scale.
4 Simulation results
Simulations show that unstacked DeepONets achieve low generalization error across dynamic-system and PDE operator-learning tasks, while sensor requirements and convergence depend on prediction horizon, input complexity, and dataset size.
- Linear dynamic system: DeepONets achieve smaller generalization and test errors than FNNs on the linear antiderivative operator, with unstacked biased models performing best.Unstacked models use fewer parameters and can be trained faster with less memory.
- Nonlinear dynamic system: Unstacked DeepONets retain smaller test MSE than stacked models for a nonlinear ODE and also predict three out-of-distribution input signals.Their training and test MSE show a tighter correlation across training and repeated random-data runs.
- Sensor requirements: The fitted sensor requirement scales as m ∝ l^-1, consistent with the theoretical result for GRF inputs with an RBF kernel, although the theoretical bound is loose.Input-function complexity also follows l ∝ m^-1 and #Bases ∝ √m in computational results.
- Error tendency and convergence: Training and test errors grow exponentially with maximum prediction time, while increasing network width first reduces error and then increases it beyond an optimal width.The later increase may reflect optimization error and could be improved by selecting a better learning rate.
- Error tendency and convergence: Test and generalization errors converge exponentially for training datasets smaller than 10^4, while generalization error converges as x^-1 for larger datasets.In the PDE problem, convergence with respect to P and input-function sample count is polynomial before saturation, and 100 samples can reach test error approximately 10^-5.
- Error tendency and convergence: The observed convergence-rate tendency follows either ln(x) or e^-x, but the authors state that it remains unexplained and requires further investigation.The convergence rate also depends on the number of input-function samples or the value of P.
5 Conclusion
The paper proposes DeepONets for learning nonlinear operators and reports small generalization errors, systematic factor studies, theoretical error dependence, and polynomial or exponential convergence. It also identifies unresolved theory concerning network-size bounds and the source of small generalization errors.
- DeepONets encode input functions and location variables separately, then merge both representations to compute outputs.
- Across four ordinary and partial differential equation problems, DeepONets achieve small generalization errors using this inductive bias.
- The study systematically varies sensor number, prediction time, input-function complexity, training-dataset size, and network size.
- The experiments show different polynomial orders and exponential error convergence with respect to training-dataset size.
- Theoretical error dependencies agree with the computational results, but network-size bounds and the mechanism behind small generalization errors remain unresolved.
A Neural networks to approximate nonlinear operators
The paper reviews universal approximation results for functionals and operators, then states compactness and continuity conditions supporting operator approximation. These results motivate representing functions through finite sensor values and approximating operator outputs at queried locations.
- A functional maps a function space into real numbers, whereas an operator maps a function space into another function space.
- A continuous nonpolynomial activation function can support universal approximation results for nonlinear continuous functionals and operators.
- The operator approximation theorem assumes compact sets in the input and output domains, a compact function family, and a nonlinear continuous operator.
- Theorems 4 and 5 require compactness of the function space; a subset of C(K) is pre-compact exactly when it is uniformly bounded and equicontinuous.
B Number of sensors for identifying nonlinear dynamic systems
The analysis establishes that sufficiently many sensor points allow the operator approximation to achieve a prescribed accuracy ε, using compactness and neural-network approximation arguments.
- Compactness argument: The proof shows that the relevant function set is pre-compact by establishing equicontinuity and uniform boundedness, then applying the Arzelà–Ascoli theorem.
- Finite-dimensional reduction: The sensor evaluations define a compact set S_m and a corresponding vector-valued function, reducing operator approximation to approximation of a finite-dimensional function.
- Approximation construction: For sufficiently large m, the sensor-induced approximation error becomes smaller than ε, after which a neural network approximates the resulting finite-dimensional function.
- Error decomposition: The operator-approximation error is bounded by the sum of the sensor discretization error and the neural-network approximation error.
- Sensor requirement: Choosing m so that c(b −a)κ(m, V )e^c(b−a) < ε is sufficient to achieve accuracy ε.
C Gaussian random field with the radial-basis function kernel
For a Gaussian random field with the radial-basis function kernel, piecewise linear interpolation of the input at m points yields a convergence rate of O(1/m^2l^2).
- Gaussian random field: The input process uses independent standard Brownian motions and a change of variable λ = lω.
- Interpolation: Applying linear interpolation on each interval enables an error estimate based on Taylor expansion.
- Convergence argument: The Borel–Cantelli lemma supports convergence of the interpolation-based approximation.
- Convergence rate: O(1/m^2l^2) convergence results when X(t) is approximated by piecewise linear interpolation with m points.