Source-linked AI summary
MIONet: Learning multiple-input operators via tensor product
Pengzhan Jin, Shuai Meng, Lu Lu
TL;DR
Existing neural operators generally accept a single function from one Banach space, limiting operators whose inputs come from different function spaces and domains. The paper proves universal approximation results for multiple-input operators, uses low-rank tensor approximation to construct MIONet, and demonstrates its performance on ODE and PDE systems, including gains from encoded prior knowledge.
Problem
Existing neural operators are designed for single-function inputs, limiting solution operators that combine functions from different Banach spaces and domains.
Method
The paper proves universal approximation theorems for multiple-input operators and proposes MIONet using multiple branch nets, a trunk net, and low-rank tensor approximation.
Results
MIONet learns solution operators for ODE and PDE systems and outperforms DeepONet in the reported ODE experiment, achieving 1.69% L2 relative error versus 2.41%.
Takeaways & Limitations
MIONet can be customized with prior knowledge such as linearity and periodicity, which further improves accuracy in the computational examples.
Takeaways & Limitations
More experiments are needed to test MIONet's performance on diverse problems.
Abstract
from arXiv · showhide
As an emerging paradigm in scientific machine learning, neural operators aim to learn operators, via neural networks, that map between infinite-dimensional function spaces. Several neural operators have been recently developed. However, all the existing neural operators are only designed to learn operators defined on a single Banach space, i.e., the input of the operator is a single function. Here, for the first time, we study the operator regression via neural networks for multiple-input operators defined on the product of Banach spaces. We first prove a universal approximation theorem of continuous multiple-input operators. We also provide detailed theoretical analysis including the approximation error, which provides a guidance of the design of the network architecture. Based on our theory and a low-rank approximation, we propose a novel neural operator, MIONet, to learn multiple-input operators. MIONet consists of several branch nets for encoding the input functions and a trunk net for encoding the domain of the output function. We demonstrate that MIONet can learn solution operators involving systems governed by ordinary and partial differential equations. In our computational examples, we also show that we can endow MIONet with prior knowledge of the underlying system, such as linearity and periodicity, to further improve the accuracy.
1 Introduction
Neural operators learn mappings between function spaces, but existing approaches were limited to single-function inputs. This work extends operator regression to multiple input functions from different Banach spaces and introduces MIONet for such problems.
- Existing SciML neural networks often solve one specific ODE or PDE instance, requiring retraining for new initial conditions.
- Neural operators instead learn solution operators that map input functions, such as initial or boundary conditions, to PDE solutions.After training, a new solution requires only a forward pass.
- Existing neural operators and their universal-approximation theory were designed for operators with a single function input from one Banach space.
- This restriction excludes operators combining inputs defined on different domains, such as initial and boundary conditions for a PDE.
- The paper develops approximation theory for multiple-input operators on products of Banach spaces and proposes MIONet to learn ODE and PDE solution operators.MIONet can incorporate prior knowledge such as linearity and periodicity to improve accuracy.
2 Approximation theory
The paper develops approximation theory for continuous operators with multiple function inputs by combining finite-dimensional representations with canonical projections in infinite-dimensional Banach spaces. The resulting equivalent representations motivate neural-network architectures, including MIONet.
- Problem formulation: The target operator maps a product of Banach spaces, K1 × ··· × Kn, continuously into an output Banach space Y.The inputs and output are typically infinite-dimensional, and the paper studies the space C(K1 × ··· × Kn, Y).
- Finite-dimensional motivation: Finite-dimensional multilinear operators can be represented through input coordinates and output tensors, motivating low-rank surrogate constructions.The approximation strategy replaces the operator with a parameterized model whose coordinate function approximates the tensor representation.
- Infinite-dimensional reduction: Canonical projections reduce compact subsets of infinite-dimensional Banach spaces to finite coordinates with arbitrarily small projection error.Schauder bases provide the coordinates, while maps ϕn and ψn decompose the projection into coordinate extraction and reconstruction.
- Main approximation theorem: Theorem 1 proves that every continuous multiple-input operator on compact input sets admits finite-dimensional continuous-function representations with approximation error below any ϵ > 0.Equivalent forms use separate input functions gi, output tensors or vectors, and summations of products.
- Network architecture: The tensor-product representation in Eq. (14) splits the input spaces into separate models and combines their outputs, leading to the proposed MIONet architecture.The paper contrasts this with architectures that concatenate all inputs before applying one standard neural network.
- Extensions and special cases: The theory extends to multiple output operators and includes single-input DeepONet approximation and function-value representations as special cases.For multiple outputs, Corollary 3 supplies analogous representations for continuous operators Gj into spaces Yj.
3 Operator regression methods
MIONet is a neural operator for multiple-input operator regression, with architectures derived from the paper’s approximation theory and low-rank formulation. It uses independent branch nets for input functions and a trunk net for output-domain locations, combining their outputs through element-wise products and summation.
- MIONet is proposed as a new neural operator for learning multiple-input operators.
- MIONet (high-rank): In the high-rank version, the trunk output represents a high-rank tensor, while the branch and trunk networks approximate separate component functions.
- Network architecture: The architecture uses independent branch nets to encode input functions and a trunk net to encode the output-domain input.
- MIONet (low-rank): The default low-rank version approximates branch and trunk functions with networks whose outputs are combined using a Hadamard product, summed, and shifted by a trainable bias.The low-rank design greatly reduces the number of parameters in the trunk net.
- Other computational details: Figure 1 depicts branch and trunk outputs with matching sizes, merged through element-wise multiplication followed by summation.
- Inference: For new input functions, MIONet produces a neural-network function that can be evaluated at arbitrary output locations without interpolation.
4 Numerical results
The experiments evaluate MIONet on ODE, diffusion-reaction, and advection-diffusion operators, comparing it with DeepONet and testing architecture customization with prior knowledge. MIONet achieves lower reported errors and periodic customization performs best for the advection-diffusion system.
- Experimental setup: The experiments learn three ODE/PDE operators using sampled input functions, fully connected branch and trunk networks, and L2 relative error across five trials.Inputs are sampled on uniform grids, and mean error and standard deviation are computed over independent training trials.
- 4.1 An ODE system: 1.69% L2 relative error for MIONet outperforms DeepONet (same size) at 2.41% and DeepONet (best) at 2.26% on the nonlinear ODE system.The best DeepONet result was selected through a depth-and-width grid search.
- 4.2 A diffusion-reaction system: MIONet has significantly lower error than both DeepONet of similar size and the best DeepONet on the diffusion-reaction system.Figure 2 compares their predictions and point-wise absolute errors against the reference solution.
- 4.3 An advection-diffusion system: For the advection-diffusion system, MIONet is modified with a linear branch for u0 and separate x and t trunk networks.The x trunk uses a periodic layer to encode the solution’s periodicity.
- 4.3 An advection-diffusion system: MIONet (periodic) obtains the smallest prediction error, outperforming MIONet, DeepONet (same size), and DeepONet (best).The compared MIONet variants and DeepONet (same size) have the same number of parameters.
5 Conclusions
The paper establishes multiple-input operator regression on products of Banach spaces and introduces MIONet as a tensor-product, low-rank neural-operator architecture. Experiments cover ODE and PDE systems, while future work calls for broader testing.
- 5 Conclusions: The paper provides universal approximation theorems for operators defined on products of multiple Banach spaces.The target maps multiple input functions from distinct Banach spaces to an output Banach space.
- 5 Conclusions: MIONet uses multiple branch nets for input functions and one trunk net for the output-function domain, based on tensor-product theory and low-rank approximation.The architecture can be customized to encode prior knowledge.
- 5 Conclusions: Three experiments demonstrate MIONet on an ODE system, a diffusion-reaction system, and an advection-diffusion system.The experiments assess solution operators for systems governed by ordinary and partial differential equations.
- 5 Conclusions: More experiments are needed to test MIONet’s performance on diverse problems.The paper also identifies POD and physics-informed extensions as future directions.
A Proofs in this study
The proofs establish approximation results for continuous multiple-input operators using finite-dimensional representations, continuous vector functionals, and neural-network approximations. They also derive related structural and complexity properties.
- Proofs: Uniform continuity and compactness are used to construct finite collections of local functionals that approximate the target operator within a prescribed error.The proof selects finite covering points and controls projection and approximation errors using the Banach-space basis constant.
- Proofs: Theorem 2 constructs continuous vector functionals and an output tensor whose composition approximates G(v1, ..., vn) within ε.The displayed bound is ||G(v1, ..., vn) − u⟨ĝ1(v1), ..., ĝn(vn)⟩|| < ε.
- Proofs: Theorem 1 follows by approximating the continuous vector functionals with neural networks and preserving the multiple-input tensor-product form.The proof connects the representation theorem to the stated universal approximation result.
- Proofs: When the operator is linear in one input, the corresponding branch representation can use an identity or linear map without losing the approximation result.The proof explicitly treats linearity with respect to v1 and notes analogous cases for other inputs.
B MIONet for finite-dimensional image space
For finite-dimensional image spaces, the general approximation result is converted into a finite-dimensional training representation and a neural-network loss with trainable weights and bias.
- B MIONet for finite-dimensional image space: The finite-dimensional image-space case is obtained by replacing the output space with a product of finite-dimensional spaces.The construction uses basis elements for the finite-dimensional domain and image representations.
- B MIONet for finite-dimensional image space: The resulting training formulation uses neural networks for the input representations together with trainable tensor weights and bias.After training, predictions are formed from the learned network representations and parameters.
C Loss function via numerical integration
The loss function uses numerical integration, with Monte Carlo integration identified as a suitable choice for high-dimensional integration.
- Numerical integration is used to evaluate the loss function for uniformly sampled points x_k on [0, 1].The sampling grid satisfies 0 = x_0 < ··· < x_m = 1.
- Monte Carlo integration is presented as one available numerical integration choice.
- For high-dimensional integration, Monte Carlo integration usually performs better.