PaperlayerView exact revision on arXiv

Source-linked AI summary

Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis

Luca Galimberti, Anastasis Kratsios, Giulia Livieri

arXiv:2210.13300v3math.DScs.LGq-fin.CP

TL;DR

Causal approximation of stochastic and infinite-dimensional dynamical systems remains limited, and quantitative links between sequential-model state and expressiveness are not well understood. The paper develops Causal Neural Operators for such settings and proves approximation results for regular operators, including stronger rates for dynamical systems in the Euclidean case.

  • Problem

    Causal neural-network approximation theory remains limited for stochastic-analysis systems, and the quantitative relationship between sequential-model state and expressiveness remains open.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs Causal Neural Operators that predict through time-specific neural filters, with a hypernetwork generating successive filters from the active filter's parameters.

    Source: §1 · p. 2

  • Results

    In the Euclidean case, neural-filter approximation rates are optimal, while CNO approximation rates for dynamical systems are super-optimal.

    Source: §6 · p. 23

  • Takeaways & Limitations

    The framework provides causal and quantitative approximation guarantees for smooth or Hölder operators between general suitable infinite-dimensional linear metric spaces.

    Source: §6 · p. 23

    The discussion presents path-dependent SDEs as a possible extension but chooses a simpler case for its first presentation.

    Source: §4.4 · p. 20

Abstract

from arXiv · show

Several non-linear operators in stochastic analysis, such as solution maps to stochastic differential equations, depend on a temporal structure which is not leveraged by contemporary neural operators designed to approximate general maps between Banach space. This paper therefore proposes an operator learning solution to this open problem by introducing a deep learning model-design framework that takes suitable infinite-dimensional linear metric spaces, e.g. Banach spaces, as inputs and returns a universal \textit{sequential} deep learning model adapted to these linear geometries specialized for the approximation of operators encoding a temporal structure. We call these models \textit{Causal Neural Operators}. Our main result states that the models produced by our framework can uniformly approximate on compact sets and across arbitrarily finite-time horizons H\"older or smooth trace class operators, which causally map sequences between given linear metric spaces. Our analysis uncovers new quantitative relationships on the latent state-space dimension of Causal Neural Operators, which even have new implications for (classical) finite-dimensional Recurrent Neural Networks. In addition, our guarantees for recurrent neural networks are tighter than the available results inherited from feedforward neural networks when approximating dynamical systems between finite-dimensional spaces.

1 Introduction

The introduction identifies gaps in causal approximation theory for stochastic dynamical systems and in quantitative understanding of how latent state size relates to sequential-model expressiveness. It proposes Causal Neural Operators (CNOs) and develops approximation guarantees for broad infinite-dimensional settings, with implications for finite-dimensional RNNs.

1 Introduction

  • Causal neural-network approximation theory remains limited for stochastic-analysis systems, while the relationship between sequential-model state and expressiveness lacks quantitative understanding.The paper singles out solution operators for non-Markovian SDEs as a significant approximation problem.Source: §1 · pp. 1–2
  • CNOs predict each output from a current input window and use a hypernetwork to generate the next time-specific neural filter while storing only one filter's parameters.The hypernetwork interpolates a finite sequence of parameter vectors rather than approximating the target operator itself.Source: §1 · p. 2
  • The CNO design supports non-recursive training, avoiding backpropagation through time while retaining a recurrent model at prediction time.The authors motivate this choice by reported practical advantages of omitting recurrence during model construction.Source: §1 · p. 3
  • Neural filters encode inputs as sparse Schauder-basis coefficients, transform them with a feed-forward network, and decode the outputs in the target space.The construction is presented for general Fréchet spaces admitting Schauder bases, not only Hilbert or Euclidean spaces.Source: §1 · pp. 3–4
  • Increasing latent dimension and network size can extend the approximable time horizon, while the paper reports more efficient causal-map approximation rates for RNNs than comparable FFNN rates.The introduction also characterizes the dynamic theorem as a quantitative guarantee for suitably regular infinite-dimensional systems.Source: §1 · p. 4

2 Preliminaries

The preliminaries establish the functional-analytic setting and neural-network conventions used by the approximation results. They introduce Fréchet spaces, differentiation and bases, then describe ReLU and PReLU feed-forward networks.

2.1 Fréchet spaces

  • A Fréchet space is a complete, metrizable, locally convex topological vector space, and every Banach space is a Fréchet space.The paper uses a metric compatible with the space's existing topology.Source: §2.1 · p. 5
  • The preliminaries use directional derivatives for maps between Fréchet spaces and require continuity of the derivative, with higher-order classes defined analogously.The paper notes that this continuity requirement is weaker than the usual Banach-space definition.Source: §2.1 · pp. 5–6
  • A Schauder basis represents every space element uniquely by a convergent series, but separability alone does not ensure that a Fréchet space has such a basis.The basis coefficient functionals are continuous and linear.Source: §2.1 · p. 6

2.2 Feedforward Neural Networks with ReLU and PReLU activation functions

  • PReLU networks can synchronize the depths of multiple realized functions, a property the paper uses in deriving its main theorem.The cited proposition states that ReLU networks can be replaced by PReLU networks with only constant-factor increases in units and weights.Source: §2.2 · p. 6

3 Main Results

The main results develop approximation guarantees for neural filters and causal neural operators, relating model complexity to regularity and geometry. They establish sequential universal approximation for causal maps, give dimension-free rates under structured Hilbert-space conditions, and describe the CNO construction and practical training approach.

3.1 Static Case: Universal Approximation

  • The guarantees require trace-class scope: an operator may be globally discontinuous yet agree on a compact set with a regular extension; without such assumptions, quantitative complexity control is unavailable.The stated obstruction is the lack of general regularity-preserving extension results between Fréchet spaces.Source: §3.1 · pp. 8, 10
  • Theorem 1 gives neural filters approximation guarantees for operators in smooth or Hölder trace classes on compact input sets, with separate encoding and approximation error tolerances.The theorem also reports model complexity as a function of the target operator’s regularity and the input and output spaces.Source: §3.1 · p. 9
  • In finite-dimensional Banach input spaces with one-dimensional output, the approximation rates match optimal ReLU-network rates, attaining the VC lower bound.The paper identifies the neural filter with a deep ReLU feedforward network in this setting.Source: §3.1 · p. 9

3.2 Dynamic Case: Sequential Universal Approximation Causal Operators

  • The proposed CNO construction trains neural filters on separate time windows and reassembles them through a central hypernetwork to causally approximate dynamical systems between infinite-dimensional spaces.The section presents construction of a finitely parameterized causal neural network approximator as its main result.Source: §3.2 · pp. 10–11
  • The dynamic framework targets causal maps whose outputs at each time depend only on past and present inputs, using finite virtual memory with horizon-dependent memory size M = O(ε^-r).The class definition imposes this memory bound for each memory-compression level ε and finite time horizon.Source: §3.2 · p. 11
  • CNOs uniformly approximate finite-virtual-memory smooth or Hölder causal maps over finite time horizons while preserving the forward flow of information through time.The theorem provides a uniform spatio-temporal estimate for the sequential model; its complexity bounds are recorded as depending on target regularity, latent-memory size, and horizon.Source: §3.2 · p. 12, Table 2 · p. 12

3.2.1 Approximation of Smooth Causal Maps Between Hilbert Spaces on Structured Compact Sets

  • Under structured Hilbert-space conditions, including smooth causal maps with zero memory compression and compact paths with exponentially decaying basis coefficients, CNOs avoid the curse of dimensionality.The corollary gives dimension-free algebraic approximation rates; for Lipschitz maps, the compact-set diameter constant must decrease faster with total error to maintain those rates.Source: §3.2.1 · pp. 13–14

3.2.2 Discussion: How the CNO could be implemented

  • The proposed training procedure first trains neural filters, then fits a hypernetwork to their sequential parameters, avoiding backpropagation through time and enabling predictions beyond training data.Transfer-learning initialization from one time step’s filter parameters reduces training iterations, but nearby parameter vectors need not represent nearby functions.Source: §3.2.2 · pp. 14–15

4 Applications to Mathematical Finance and Stochastic Analysis

This section applies Causal Neural Operators to static pricing functionals on a non-Banach Fréchet space and to causal approximation of stochastic solution maps, including SDEs with varied initial conditions and time horizons. It also provides technical references and conditions for simultaneous approximation of SDEs with different initial conditions.

4.1.1 Functionals on a Fréchet Space which is in Not Banach: Forward Rate Curves

  • The forward-curve pricing functional is locally Lipschitz on L1_loc, a Fréchet space that is not Banach, placing it within the paper’s approximation setting.The example concerns derivative pricing in power markets, where forward prices are represented by locally integrable functions.Source: §4.1.1 · pp. 15–16

4.1.2 Efficient Approximation Rates for Pricing with Smooth Rapidly Decaying Functions

  • On compact sets of rapidly decaying smooth functions, the pricing functional can be efficiently approximated using the paper’s guarantee for exponential decay of Hermite coefficients.The decay condition is interpreted as jointly expressing tail decay and smoothness; the argument restricts the functional to L2(R), where it remains Lipschitz.Source: §4.1.2 · p. 16

4.2 Dynamic Examples: Solution Operators of Stochastic Differential Equations

  • The paper applies its results to stochastic-analysis solution operators and illustrates their use for numerical SDE solutions without requiring strong structural assumptions on process evolution.It also points to processes with jumps as a broader application area.Source: §4.2 · p. 16

4.3 A primer on Wiener Chaos

  • Wiener chaos supplies an orthogonal decomposition of L2(Ω, F_t, P), providing a basis for representing random variables in the Brownian filtration.The construction uses iterated stochastic integrals and Hermite polynomials.Source: §4.3 · pp. 17–18

4.4 Simultaneous Approximation of SDEs with Different Initial Conditions using CNOs

  • A single CNO can simultaneously approximate SDE solution maps for stochastic initial conditions across different initial times and arbitrarily long time horizons on compact path spaces.The result applies under the stated growth and Lipschitz conditions; sufficiently differentiable drift and diffusion coefficients yield improved approximation rates, while jumps are accommodated only at fixed time-grid points. Path-dependent coefficients are discussed as a possible extension, not treated in this presentation.Source: §4.4 · pp. 19–20
  • The subsection refers readers to Definition 8.Source: §4.4 · p. 20
  • It recalls Definition 6 while specifying notation for ˆf_t^i through a composition of maps.The displayed composition is partially garbled in the supplied passage.Source: §4.4 · p. 20
  • The precise conditions are formalized in [92, Assumption 3.7].This reference is given in the subsection on simultaneous approximation of SDEs with different initial conditions.Source: §4.4 · p. 20

5 The Benefit of Causal Approximation: Super-Optimal Approximation Rates for Causal Maps

The section compares causal and feedforward approximation for causal maps, then relates Causal Neural Operators to recurrent networks. Its results show an approximation-rate advantage for CNOs and connect that architecture to quantitative questions about recurrent models.

5 The Benefit of Causal Approximation: Super-Optimal Approximation Rates for Causal Maps

  • The comparison is motivated by the lack of lower bounds for the causal map class, despite known lower bounds for unrestricted Lipschitz maps.The cited ReLU upper bounds match those unrestricted-map lower bounds.Source: §5 · p. 21
  • Quantitative approximation of nonlinear dynamical systems and the role of recurrent models’ latent-state dimension remain open questions that the section addresses through Theorem 2.The discussion contrasts implementable recurrent models with unrestricted sequences of networks that may have unrelated parameters.Source: §5.1 · p. 21
  • In Euclidean spaces, CNOs reduce to a simple class of RNNs, with weights and biases determined recursively by a hypernetwork rather than the input sequence.The section identifies this as essentially an Elman RNN with multi-layer networks.Source: §5.1 · pp. 21–22
  • For a causal map defined by z(t) = G(z_t, z(t−1)), the section compares CNO and deep ReLU FFNN approximation on [0, 1]^T.The recurrence starts from z(0) = 0, and the target is the final output z(T).Source: §5.1 · p. 22
  • CNOs achieve super-optimal uniform approximation rates for the studied causal maps, without a dimensionality curse in the number of time steps.The comparison is against deep ReLU FFNNs, which are described as cursed by that dimension.Source: §5.1 · pp. 22–23

6 Conclusion

The paper presents causal and static approximation guarantees for smooth or Hölder operators between Fréchet spaces, and reports CNO approximation results for stochastic-analysis operators and optimality properties in Euclidean settings.

6 Conclusion

  • CNOs approximate smooth or Hölder nonlinear operators between Fréchet spaces in both static and causal settings.The causal setting includes temporal structure; the static setting does not.Source: §6 · p. 23
  • The CNO model approximates solution operators and infinite-dimensional dynamical systems arising in stochastic analysis.Source: §6 · p. 23
  • In Euclidean settings, neural-filter approximation rates are optimal, while CNO rates are super-optimal for dynamical systems under parameter-count optimality.The stated optimality notion concerns the number of parameters needed to approximate maps in a broad class, as in.Source: §6 · p. 23
  • Further work is proposed on optimizing CNOs for stochastic filtering under additional structural conditions and on applying the results to robust finance.Source: §6 · p. 23

A Background material for proofs

The appendix collects background used in deriving the main results but not needed to formulate them, including neural-network, dimension-reduction, and generalized-inverse material.

A Background material for proofs

A.1.1 DNN Approximation for Smooth and Hölder Functions

This appendix section reviews neural-network approximation results for smooth and Hölder functions, including network size bounds, a smooth-function norm, and a parallelization of results across width and depth.

A.1.1 DNN Approximation for Smooth and Hölder Functions

  • ReLU FFNNs can approximate f in Cs([0, 1]^d) with nearly optimal error O(∥f∥Cs N^-2s/d L^-2s/d), using width O(N log(N)) and depth O(L log(L) + d).The cited theorem gives this result for functions in Cs([0, 1]^d).Source: §A.1.1 · p. 23
  • The Cs norm is defined as the maximum L∞ norm of derivatives ∂^αf for multi-indices with |α| ≤ s.Source: §A.1.1 · p. 23
  • For the cited ReLU approximation theorem, width and depth contribute symmetrically because both N and L have error exponent −2s/d.The passage states the width and depth are not privileged relative to one another.Source: §A.1.1 · p. 23
  • The section also states a Hölder-function approximation result with width depending on d and N and depth 11L + C2, but the supplied passage does not give its error bound.The result assumes Hölder order α ∈ (0, 1] and constant λ > 0.Source: §A.1.1 · p. 24
  • The smooth-function theorem excerpt specifies constants C1 = 16 and C2 = 18 for p ∈ [1, ∞), and C1 = 3d + 3 and C2 = 18 + 2d for p = ∞.Source: §A.1.1 · p. 24

A.1.2 Efficient parallelization of ReLU neural networks

The section reviews parallelization of neural networks with differing depths and defines a condition on activations that supports the stated parallelization result.

A.1.2 Efficient parallelization of ReLU neural networks

  • The background defines network depth, layer widths, parameter count, and the affine maps used in a network skeleton’s realization.Source: §A.1.2 · p. 24
  • A c-identity activation has a two-layer network realization of the identity with first-layer width at most c.The definition requires c ≥ 2.Source: §A.1.2 · p. 24
  • ReLU satisfies the 2-identity requirement using the network I specified in the section.Source: §A.1.2 · p. 24
  • For activations meeting the c-identity requirement, proposition 1 gives a parallelization pI for a collection of network skeletons.The excerpt identifies the parallelization as combining φ1, …, φn.Source: §A.1.2 · p. 24

A.1.3 Memory Capacity of Deep ReLU regressor

This material gives quantitative tools for memorizing finite data with deep ReLU networks and supporting approximation arguments in linear spaces.

A.1.3 Memory Capacity of Deep ReLU regressor

  • A measure-space aspect ratio is total mass divided by the minimum point mass, and larger ratios make finite Euclidean datasets logarithmically harder to memorize.The text attributes this relationship to.Source: §A.1.3 · p. 24
  • The ReLU memorization lemma interpolates arbitrary values at distinct inputs and supplies width, depth, and nonzero-parameter complexity estimates.The supplied passages do not preserve the full formulas for those estimates.Source: §A.1.3 · p. 24
  • A locally convex space has the bounded approximation property when finite-rank operators from an equicontinuous net converge pointwise to the identity.Barreled locally convex spaces with a Schauder basis have this property.Source: §A.2 · p. 25
  • A finite-dimensional vector space has only one vector-space topology up to homeomorphism.Source: §A.2 · p. 25

B Proofs

The proofs establish the CNO approximation result through continuity and smoothness controls, finite-dimensional neural approximation, and a recurrent memorizer construction that recursively updates time-specific filter parameters. They also derive a width-based corollary and complexity bounds.

B.1 Proof of Lemma 1

  • B.1 shows that composing a Ck-Dir operator with continuous linear maps preserves Ck stability, by the chain rule and continuity of its partial derivatives.The maps ˜I and ˜P are Ck-Dir, so ˜P ◦ f ◦ ˜I is Ck in the usual finite-dimensional sense.Source: §B.1 · p. 25
  • B.1 proves that a family of maps satisfying uniform continuity with a shared input-distance condition has a common modulus of continuity.Equicontinuous families of continuous linear maps between Fréchet spaces satisfy the lemma’s assumption.Source: §B.1 · pp. 25–26

B.2 Proof of Theorem 1

  • B.2 reduces the target operator to a finite-dimensional surrogate, approximates it with a ReLU MLP, and reconstructs an infinite-dimensional prediction while controlling errors at each step.The construction projects inputs to a finite-dimensional subspace and interprets MLP outputs as coefficients in a Schauder basis.Source: §B.2 · p. 26
  • B.2 handles the Hölder trace-class case by changing the input-dimension choice and using Theorem 4; Lipschitz pre- and post-composition preserve α-Hölder regularity.The approximation estimate uses Theorem 4 instead of Theorem 3.Source: §B.2 · p. 29

B.3 Proof of Corollary 1

  • The corollary proof uses linear and Kolmogorov widths; these notions coincide in Hilbert spaces, and inclusion yields d_i(X_tn, E_tn) ≤ d_i(Z_n, E_tn) [91].The inequality follows when X_tn is contained in the relevant subspace used to bound Z_n.Source: §B.2 · pp. 29–30
  • O(ε^-3/2 log(1/ε)^3) bounds the non-zero trainable parameters for the CNO filters in the corollary’s construction.The proof also bounds hypernetwork complexity in terms of P([d]) and the time horizon I.Source: §B.2 · p. 31

B.4 The Dynamic Weaving Lemma

  • Memorizing N points requires between N and N^2 trainable parameters, whereas uniform approximation over a hypercube can require exponentially many neurons and [52].The Dynamic Weaving Lemma uses this distinction to combine neural filters on distinct time windows; its proof packs separated codes and constructs a ReLU memorizer.Source: §B.4 · p. 31

B.5 Proof of Theorem 2

  • Theorem 2’s proof applies Theorem 1 at each time point, then uses a memorizer to update filter parameters recursively while projecting away auxiliary codes and controlling errors.The auxiliary codes ensure separation of parameter vectors, even when the original vectors coincide.Source: §B.5 · p. 33

C Technical Lemmata

The technical lemmas show how two Fréchet spaces with Schauder bases can be combined while retaining a Schauder basis in the product topology.

C Technical Lemmata

  • The product of the two spaces carries a Schauder basis formed by interleaving their individual basis vectors.The construction alternates vectors from each factor.Source: C Technical Lemmata · p. 34
  • With the product topology, the combined space is a complete, locally convex topological vector space.The text gives a compatible metric and a family of seminorms inducing the topology.Source: C Technical Lemmata · p. 34
  • Every element of the product has a unique expansion in the constructed basis.The proof establishes convergence of the expansion and uniqueness by projecting onto a factor space.Source: C Technical Lemmata · pp. 34–35

D Additional Background Material

This appendix collects background on generalized inverses and verifies that a metric induced on finite coordinate vectors preserves their standard topology.

D Additional Background Material

  • For a separable Fréchet space with a Schauder basis, the metric induced on R^n has the standard topology.The proof compares convergent sequences under the induced metric and the Euclidean norm.Source: D Additional Background Material · pp. 35–36
  • The appendix presents these generalized-inverse properties as background results drawn from.Source: §D.2 · p. 36
  • For an increasing function, the generalized inverse is increasing and left-continuous wherever finite, and it recovers the original argument under strict increase.The listed properties also describe endpoint and right-continuity behavior.Source: §D.2 · p. 36

Related papers

Loading 2210.13300v3…