Source-linked AI summary
Deep Operator Network Approximation Rates for Lipschitz Operators
Christoph Schwab, Andreas Stein, Jakob Zech
TL;DR
Approximating operators between infinite-dimensional Hilbert spaces requires addressing the curse of dimensionality, and prior rate results often rely on holomorphic dependence. This paper uses linear basis-based encoders and decoders with neural approximators of coordinate maps to establish expression rates for lower-regularity operators. The supported rates are algebraic under stated regularity and sparsity conditions, while some nonstandard architectures carry a floating-point-operation cost that grows exponentially with height.
Problem
For operators between infinite-dimensional spaces, efficient approximation must overcome the curse of dimensionality, while some prior neural emulation rates rely on holomorphic dependence.
Source: §1 · p. 2Method
The framework represents the operator with linear encoders and decoders based on stable biorthogonal bases and a neural approximator of the coordinate map.
Source: §2 · p. 4, §8 · p. 25Results
The expression-rate bounds are algebraic when input and output regularity and sparsity, the operator mapping properties, and super-expressive neural approximations satisfy the stated conditions.
Source: §8 · p. 25Takeaways & Limitations
The framework covers operator-network constructions using bases such as Fourier and Karhunen–Loève representations, with sampling or quadrature aliasing errors requiring separate accounting in those cases.
Source: §8 · p. 25For NestNets of height h ≥ 2, floating-point operations grow exponentially in h, so avoiding the curse by neuron count does not remove it by operation count.
Source: §5 · p. 11
Abstract
from arXiv · showhide
We establish universality and expression rate bounds for a class of neural Deep Operator Networks (DON) emulating Lipschitz (or H\"older) continuous maps $\mathcal G:\mathcal X\to\mathcal Y$ between (subsets of) separable Hilbert spaces $\mathcal X$, $\mathcal Y$. The DON architecture considered uses linear encoders $\mathcal E$ and decoders $\mathcal D$ via (biorthogonal) Riesz bases of $\mathcal X$, $\mathcal Y$, and an approximator network of an infinite-dimensional, parametric coordinate map that is Lipschitz continuous on the sequence space $\ell^2(\mathbb N)$. Unlike previous works ([Herrmann, Schwab and Zech: Neural and Spectral operator surrogates: construction and expression rate bounds, SAM Report, 2022], [Marcati and Schwab: Exponential Convergence of Deep Operator Networks for Elliptic Partial Differential Equations, SAM Report, 2022]), which required for example $\mathcal G$ to be holomorphic, the present expression rate results require mere Lipschitz (or H\"older) continuity of $\mathcal G$. Key in the proof of the present expression rate bounds is the use of either super-expressive activations (e.g. [Yarotski: Elementary superexpressive activations, Int. Conf. on ML, 2021], [Shen, Yang and Zhang: Neural network approximation: Three hidden layers are enough, Neural Networks, 2021], and the references there) which are inspired by the Kolmogorov superposition theorem, or of nonstandard NN architectures with standard (ReLU) activations as recently proposed in [Zhang, Shen and Yang: Neural Network Architecture Beyond Width and Depth, Adv. in Neural Inf. Proc. Sys., 2022]. We illustrate the abstract results by approximation rate bounds for emulation of a) solution operators for parametric elliptic variational inequalities, and b) Lipschitz maps of Hilbert-Schmidt operators.
1 Introduction
Deep neural networks are studied as scientific-computing approximators, but operator emulation between infinite-dimensional spaces must contend with the curse of dimensionality. This paper develops expression-rate results for Lipschitz or Hölder operators using operator-network architectures and identifies settings where algebraic rates are available.
1 Introduction
- Operator emulation between infinite-dimensional spaces must overcome the curse of dimensionality; relevant examples include parameter-to-solution maps and digital twins.Prior low-regularity results cited here can be mean-square or high-probability bounds, while other cited rates rely on holomorphic dependence.Source: §1 · p. 2
1.1 Previous work for operator networks
- Earlier operator-network rate analyses addressed holomorphic maps, while lower bounds for Lipschitz maps indicate PCA-nets with standard deep feedforward approximators generally cannot avoid the curse of dimensionality or achieve algebraic convergence [38] [36].The paper's approach uses super-expressive activations or more highly connected nonstandard architectures, so its results do not contradict those lower bounds [58] [64].Source: §1.1 · p. 3, §1.2 · p. 3
1.2 Contributions
- The contribution also includes a universal approximation theorem for continuous operators, with uniform convergence on any compact subset of the input space.Source: §1.2 · p. 3
- The paper proves algebraic expression-rate bounds for Lipschitz or Hölder maps between infinite-dimensional separable Hilbert spaces, with rates free of the curse of dimensionality on higher-regularity inputs.The bounds use finite-parametric neural-operator surrogates built from countably-parametric maps.Source: §1.2 · p. 3
1.4 Layout
- The paper applies its abstract results to solution maps for parametric elliptic variational inequalities and to Lipschitz maps of Hilbert-Schmidt operators.Source: §1.4 · p. 4
2 Setting
The operator is represented through linear encoders and decoders that convert between Hilbert-space elements and coefficient sequences, with a coordinate map acting on those sequences. Rate analysis is formulated on input and output subspaces whose basis coefficients have weighted summability.
2 Setting
- The operator is factored through an infinite-parametric map on ℓ2(N), using a linear encoder from the input space and a linear decoder to the output space.The encoder analyzes inputs into coefficients; the decoder synthesizes outputs, and the representation systems may be Riesz bases rather than orthogonal bases.Source: §2 · p. 4
2.1 Encoders and Decoders
- The framework admits Riesz-basis encoders and decoders, including Fourier, wavelet, and Karhunen–Loève bases.For orthonormal bases, the dual basis coincides with the basis; general Riesz bases have corresponding unique dual bases.Source: §2.1 · p. 5
2.2 Smoothness Scales X s, Yt
- Rate analysis is conducted on input and output subspaces with extra regularity expressed as weighted summability of their basis-expansion coefficients.The paper introduces smoothness-indexed Hilbert spaces for this coefficient-decay condition; examples include Sobolev, Besov, and Cameron–Martin spaces.Source: §2.2 · pp. 5–6
3 Universality
The operator-net architecture is universal for continuous maps between separable Hilbert spaces, with convergence uniform on compact input subsets. This universality complements rate results that rely on additional regularity.
3 Universality
- The paper’s rate analysis begins with universality for any continuous G, while its expression-rate results concern operators with additional regularity.Source: §3 · p. 6
- Operator nets with feedforward approximators using activations from the admissible class are universal for continuous maps between separable Hilbert spaces.The resulting sequence converges uniformly on every compact subset of the input space.Source: §3 · p. 6
- The theorem implies universality for maps between Hilbertian Sobolev spaces Hs(D) and Hs′(D′), for s, s′ ∈ R.Source: §3 · p. 6
4 Dimension Truncation
Dimension truncation turns the infinite-dimensional operator representation into finite-parametric surrogates. The error bounds use weighted coefficient regularity and separate output truncation from componentwise input truncation.
4 Dimension Truncation
- The analysis retains finitely many encoded input parameters and output components, with higher regularity characterized through weighted summability of coefficient sequences.Source: §4 · p. 7
4.2 Decoding Dimension Truncation
- Under Assumption 4.1, output truncation has an error bound that decays with the number N of retained output components.Source: §4.2 · p. 7
4.3 Encoding Dimension Truncation
- For component j, the input truncation uses its own index Mj, yielding a finite-dimensional surrogate with an error bound for arbitrary truncation indices.Source: §4.3 · p. 8
- When all components use a common truncation index M, the input-truncation bound specializes to the equal-index case.Source: §4.3 · p. 8
5 Deep Operator Surrogates
The finite-dimensional coordinate maps are replaced by neural-network surrogates to form operator approximations. The rate analysis combines input and output truncation with scalar-network approximation, under assumptions on the operator and the approximator architecture.
5 Deep Operator Surrogates
- The scalar-network approximation assumption covers Lipschitz functions on finite-dimensional cubes, with parameter counts depending on dimension and approximation tolerance.Source: §5 · p. 10
- NestNets with ReLU activations obtain curse-free approximation rates in neuron count, but their forward-pass floating-point operations grow exponentially with height.Source: §5 · pp. 10–11
- A FLES network approximates Lipschitz functions using floor, exponential, and binary-step activations, with a parameter count bounded by O(d(1 + log(d) + |log(ε)|)).Source: §5 · p. 11
- Theorem 5.7 bounds the parameters needed for an ε-consistent neural operator surrogate of the coordinate map under Assumptions 4.1 and 5.1.The construction first selects truncation indices, then substitutes neural-network surrogates for the coordinate maps.Source: §5 · p. 13
- The theorem’s operator-level complexity bound combines truncation with scalar surrogates for the component maps, whose parameter counts depend on each Mj and εj.Source: §5 · pp. 14–15
6 Discussion and Extensions
The discussion extends the operator-emulation result to finite compositions of Lipschitz maps and to Hölder-continuous operators. The latter have weaker input-truncation behavior and modified parameter bounds.
6.1 Compositions of Lipschitz Maps
- Finite compositions of Lipschitz maps remain Lipschitz, with the composition constant bounded by the product of the factor-map constants.Source: §6.1 · p. 17
- For a composition, only the last factor must satisfy Assumption 4.1; earlier factors need only be Lipschitz.Source: §6.1 · p. 17
6.2 H¨older Continuous Maps
- For γ-Hölder operators, output-truncation rates retain their form, but input-truncation error worsens and the parameter bound changes.The stated exponent scaling reflects both the reduced input-truncation rate and deteriorating scalar-network approximation rates.Source: §6.2 · pp. 17–18
7 Examples
The examples illustrate how the framework covers operators with Lipschitz or Hölder regularity, including cases without greater regularity. They consider elliptic variational inequalities and maps on Hilbert-Schmidt operators.
7 Examples
- The examples focus on Lipschitz operators that generically have no regularity beyond Lipschitz or Hölder continuity.The paper notes that holomorphic maps are also covered, but does not use them as illustrative examples.Source: §7 · p. 18
- Examples of elliptic variational inequalities include obstacle problems, optimal stopping, and optimal control of differential operators.The paper cites obstacle problems in mechanics as one setting.Source: §7 · p. 18
- The framework accommodates nonlinear parametric elliptic variational inequalities and Lipschitz maps of Hilbert-Schmidt operators.The latter act between separable Hilbert spaces.Source: §7 · p. 18
7.1 Elliptic Variational Inequalities
The examples apply the operator-network framework to elliptic variational inequalities, whose solution maps are Lipschitz under stated conditions. They develop an obstacle-to-solution example on the torus and derive approximation-rate bounds using wavelet encoding.
7.1 Elliptic Variational Inequalities
- Coefficient-to-solution maps for elliptic variational inequalities are Lipschitz, but not better, even for smooth obstacles.The paper connects this regularity to DON emulation and bases its rate analysis on classical Lipschitz stability under data perturbations.Source: §7.1 · p. 18
7.1.1 Abstract Setting
- Under strong monotonicity, Lipschitz continuity, and closed convex constraint sets, the variational inequality has a unique solution with Lipschitz dependence on its data.The stability estimate relates solution differences to perturbations in operators, source terms, and constraint sets.Source: §7.1.1 · pp. 18–19
7.1.2 Elliptic Variational Inequalities on the Torus
- On the torus, varying the obstacle defines an obstacle-to-solution operator for a variational inequality with a unique solution.The example fixes the operator and source term while varying the obstacle.Source: §7.1.2 · p. 19
7.1.3 Wavelet Encoding
- Wavelet bases represent the relevant Besov and Sobolev smoothness spaces, providing the encoding used in the torus example.The construction uses compactly supported, scaled and translated wavelets, with smoothness available by choosing enough vanishing moments.Source: §7.1.3 · p. 20
7.1.4 Neural Network Approximation Rates of parametric EVIs
- The example derives finite-parametric neural approximations and parameter-count bounds for obstacle solution maps, while fixed input and output truncations can yield outputs outside the obstacle constraint set.A postprocessing step is described to obtain a finite-dimensional approximation that lies in the truncated constraint set.Source: §7.1.4 · p. 22
7.2 Expression Rates for Lipschitz Maps of Hilbert-Schmidt Operators
The section treats nonlinear maps on Hilbert-Schmidt operators as maps between separable Hilbert spaces. It develops a singular-value functional-calculus example and reports more favorable DON emulation rates for that class than for the general case.
7.2.1 Hilbert-Schmidt Operators
- Hilbert-Schmidt operators form a separable Hilbert space, allowing the abstract framework to be applied with the operator space as both input and output.They are characterized by square-summable singular values.Source: §7.2 · p. 23, §7.2.1 · p. 23
7.2 Expression Rates for Lipschitz Maps of Hilbert-Schmidt Operators
- For a particular class of nonlinear maps on Hilbert-Schmidt operators, the paper shows better DON emulation rates than in the general case.The class is constructed through functional calculus using a scalar Lipschitz function.Source: §7.2.2 · p. 23
7.2.2 Singular Value Lipschitz Functional Calculus
- For Lipschitz f with Lipschitz constant Lf > 0, G(f) maps S2(H1,H2) to itself Lipschitzly [1, Theorem 4.2]; without further assumptions on f, G(f)(A) may lie outside S2(H1,H2) for A in S2(H1,H2).Source: §7.2.2 · p. 23
7.2.3 DNN Emulation Rates
- For the singular-value maps, the construction truncates the operator and approximates the scalar Lipschitz function to obtain a finite-parameter DNN surrogate.The analysis fixes input and output truncation by N terms simultaneously.Source: §7.2.3 · p. 24
8 Conclusions
The conclusions present DON expression-rate bounds for Lipschitz maps using stable basis-based linear encoders and decoders, with superexpressive approximators. The framework covers several operator-network constructions, but its bounds and linear decoding impose scope constraints.
8 Conclusions
- The DON framework uses stable biorthogonal bases for linear encoding and decoding, with superexpressive activations or nonstandard architectures for approximating low-regularity coordinate maps.The conclusions cite Nest-Net as one nonstandard architecture and note that concrete encoders may require accounting for sampling or quadrature aliasing errors.Source: §8 · p. 25
- The framework includes multiresolution bases such as wavelets and therefore comprises multi-fidelity operator networks.The conclusions also describe possible extensions to separable Banach spaces and operator-to-function maps in inverse problems.Source: §8 · p. 25
- The rate analysis uses linear encoders, whereas adaptive nonlinear encoding could achieve the same rates for larger input classes.Extending the framework to such encoders is left for future work.Source: §8 · p. 25
- The analysis yields algebraic, curse-of-dimensionality-free consistency orders on higher-regularity inputs for finite-parametric surrogates.The conclusions identify regularity and sparsity, mapping properties, and superexpressivity as conditions behind the bounds.Source: §8 · p. 25
- Linear decoding can poorly capture Lipschitz manifolds without further assumptions; nonlinear, data-dependent decoding could offer greater expressivity.The stated linear decoding rates apply when the operator range lies in the specified smoothness space.Source: §8 · pp. 25–26
A Proof of Universality
The proof constructs approximants that converge uniformly to the operator on every compact subset of the input space. It uses compactness and continuity to control the finite-coordinate approximations involved in that construction.
A Proof of Universality
- The proof constructs a sequence of approximants converging uniformly to the operator on each compact input set.The constructed sequence is independent of the compact set used to establish convergence.Source: A Proof of Universality · p. 26
- Compactness of the input set and uniform continuity of the operator let input approximations control output error.For any ε > 0, the proof chooses δ > 0 so that inputs within δ have outputs within ε.Source: A Proof of Universality · p. 27
- The encoded compact input set, together with its finite-coordinate truncations, forms a compact subset of ℓ2(N).This compactness supports the proof's subsequent truncation and approximation steps.Source: A Proof of Universality · p. 27
- Continuity of the coordinate functions and the truncation bounds are used to conclude convergence of the constructed approximants.The proof concludes by applying the bounds for input and output truncation and then letting ε be arbitrary.Source: A Proof of Universality · p. 28