PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual neural networks overcome the curse of dimensionality for semilinear heat equations

Ilkhom Mukhammadiev, Diyora Salimova

arXiv:2609.03626v1math.NAcs.LGcs.NAmath.APmath.PR

TL;DR

The paper addresses the limited rigorous evidence for ResNets approximating solutions of nonlinear high-dimensional PDEs. It represents a deterministic realization of an MLP estimator with an accumulator ResNet and proves polynomial parameter bounds for semilinear heat equations under specified assumptions. For ridge-sum data and admissible sigmoidal activations, it also gives an explicit parameter bound.

  • Problem

    Rigorous polynomial-parameter approximation results for high-dimensional PDEs are comparatively limited for ResNets in the nonlinear setting.

    Source: §1.1 · pp. 1–2

  • Method

    The proof represents one deterministic MLP-estimator realization with a ResNet whose shortcuts carry the spatial variable and accumulator while residual updates add estimator summands.

    Source: §4.1 · p. 15, §7 · p. 19

  • Results

    For every ξ > 0, the ridge-sum setting yields the bound Cξd^(4+ξ)ε^-(3+ξ) on the number of parameters.

    Source: §7 · p. 19

  • Takeaways & Limitations

    Under the stated hypotheses, ResNets provide deterministic polynomial-size approximations for the considered semilinear heat-equation solutions.

    Source: §1.1 · p. 2, §7 · p. 19

    The concrete ridge-sum result does not cover general radial or generic Lipschitz initial data, and broader structured classes are left for future work.

    Source: §4 · p. 13

Abstract

from arXiv · show

Rigorous results show that feedforward neural networks can overcome the curse of dimensionality in the numerical approximation of high-dimensional partial differential equations (PDEs), but comparatively little is known about residual neural networks (ResNets) in the nonlinear PDE setting. We prove that ResNets overcome the curse of dimensionality in the numerical approximation of solutions of semilinear heat equations with globally Lipschitz continuous, gradient-independent nonlinearities: under polynomial growth and network approximability hypotheses on the PDE data, there exist $\eta\in(0,\infty)$ and ResNets $\Psi_{d,\varepsilon}$, $d\in\mathbb{N}$, $\varepsilon\in(0,1]$, with at most $\eta d^{\eta}\varepsilon^{-\eta}$ parameters whose realizations approximate the solution in dimension $d$ with an $L^2$-error of at most $\varepsilon$. The proof represents one deterministic realization of a multilevel Picard estimator by a ResNet whose shortcut connections transmit the spatial variable and a scalar accumulator, while the residual branches successively add the summands of the estimator. For ridge-sum initial conditions, admissible sigmoidal activations, and globally Lipschitz truncations of the nonlinearity, we obtain, for every $\xi>0$, the explicit bound $C_\xi d^{4+\xi}\varepsilon^{-(3+\xi)}$ on the number of parameters.

1 Introduction

The introduction establishes a rigorous polynomial-parameter approximation result for ResNets applied to semilinear heat equations under suitable data and nonlinearity hypotheses. It places the result in the context of prior PDE approximation work and motivates an accumulator-based ResNet construction.

1.1 The problem and the main result

  • ResNets achieve polynomial parameter growth in dimension and inverse accuracy for semilinear heat-equation approximation under the stated hypotheses.The result is an expressivity guarantee: it establishes existence, not that a prescribed training procedure finds such a network.Source: §1.1 · p. 2

1.2 Deep learning for high-dimensional PDEs and rigorous approximation results

  • For semilinear nonlinear equations, prior MLP results provide polynomial-cost approximations, while prior feedforward representation results include identity-representing activation requirements.The article's ResNet result removes that activation requirement for its gradient-independent semilinear heat-equation setting.Source: §1.2 · pp. 2–3
  • The result does not imply polynomial-time training, and its networks need not have uniformly bounded depth.The cited sampling-complexity lower bound is consistent with an expressivity guarantee but does not establish a polynomial-time training algorithm.Source: §1.2 · p. 3

1.3 Residual networks and their approximation theory

  • The construction accumulates MLP-estimator summands through residual updates, avoiding an identity-representing activation and parallelization of different network widths.The spatial variable and scalar accumulator pass through shortcuts; admissible activations include logistic and tanh.Source: §1.3 · p. 4

2 A calculus of feedforward and residual networks

This section sets out the feedforward and residual network notation and the realization rules used in the paper. ResNet concatenation adds parameter counts, while a one-coordinate lift supports the later accumulator construction.

2.1 Feedforward neural networks (FNNs)

  • A feedforward network realization applies the activation componentwise between affine layers and uses a final affine layer for its output.The section also defines network depth, architecture, input/output dimensions, and parameter count.Source: §2.1 · p. 5

2.2 Residual neural networks (ResNets)

  • A ResNet realization updates its state by adding each residual-branch output to the shortcut-transformed state.When shortcuts are identities, this becomes an explicit Euler-type additive update.Source: §2.2 · p. 6
  • Concatenating ResNets composes their realizations and adds their parameter counts.This additivity supports building larger networks from successive residual updates.Source: §2.2 · p. 6
  • The one-coordinate lift enlarges each layer by one coordinate while preserving the original ResNet computation in the other coordinates.The additional coordinate can carry an unchanged scalar through the network.Source: §2.2 · p. 7

3 Semilinear heat equations and MLP approximations

The section sets up the semilinear heat-equation problem and its multilevel Picard approximation, then identifies stability and balancing estimates used to represent the approximation with a ResNet.

3 Semilinear heat equations and MLP approximations

  • The setting assumes globally Lipschitz nonlinearities, polynomial growth bounds, and perturbed data that can later be chosen as network realizations.The original PDE data and perturbed data are treated together under the same bounds.Source: §3 · p. 7
  • Time reversal identifies the fixed-point solution at time zero with the PDE solution at time T, while allowing continuous initial data smoothed by heat flow.The MLP estimator is defined to approximate the fixed-point solution; a deterministic realization is later represented by a ResNet.Source: §3 · p. 8
  • Because the nonlinearity is gradient-independent, its operator acts pointwise, which the MLP recursion exploits.The MLP recursion is the object whose estimator realization is subsequently represented by a ResNet.Source: §3 · p. 8

3.1 Two quantitative inputs

  • The proof’s quantitative inputs are moment and PDE-stability bounds, an MLP stability bound, and a balancing estimate for the MLP error factor.The balancing estimate chooses N for a prescribed ε while balancing the error factor against a recursive representation factor.Source: §3.1 · p. 9

4 ResNet representation of MLP estimators

This section shows how fixed MLP-estimator realizations can be represented by ResNets with polynomially bounded parameter counts, using accumulator blocks and a deterministic recursion. It derives a parameter recursion for the construction and bounds its normalized growth geometrically.

4 ResNet representation of MLP estimators

  • The construction requires no activation network that exactly represents the identity, unlike the cited feedforward arguments.Residual updates are combined by concatenation, with no identity subnetwork inserted between them.Source: §4 · p. 10
  • Admissible sigmoidal activations yield O(δ^-1)-parameter surrogates for globally Lipschitz truncations and ridge-sum data under the stated uniform profile conditions.The truncation surrogates have uniform Lipschitz bounds; the ridge-sum class allows a number of profiles independent of dimension and verifies the data-approximation setting with p = κ = 1.Source: §4 · pp. 11–13
  • The ridge-sum class covers shallow-network and single-index data, but not generic Lipschitz or radial data; broader structured classes remain for future work.Bounded Lipschitz truncations of basket payoffs are covered when profile bounds are dimension-uniform; comparison with the original payoff requires a separate Gaussian-tail estimate.Source: §4 · p. 13

4.1 ResNet representation of an MLP sample

  • A fixed MLP-estimator realization is represented by a ResNet that preserves the spatial input and accumulates each estimator summand in a scalar coordinate.The updates are concatenated without common-width padding; the final scalar readout yields the represented estimator.Source: §4.1 · pp. 13–15

4.2 The parameter recursion

  • Level-n accumulators admit deterministic parameter bounds uniform over their indices, times, and sampled increments, and the scalar-output ResNet inherits the bound.Updates involving g and f composed with lower-level estimators contribute costs depending on surrogate sizes and lower-level accumulator bounds; one-hidden-layer g-surrogates permit a sharper coefficient.Source: §4.2 · pp. 15–16
  • Concatenation yields the parameter-count recursion (43), with the identity accumulator in the base case using at most Cd^2 parameters.The two root blocks together also use at most Cd^2 parameters, an overhead absorbed by Bd,δ.Source: §4.2 · p. 16
  • Normalizing p_n,M as q_n,M = p_n,M/(M^n B_d,δ) gives q_n,M ≤ C(1+c_1)(1+C)^(n−1), establishing geometric growth.The bound follows from the normalized recursion and base case; the universal constant is adjusted to obtain (44).Source: §4.2 · p. 16

5 Main theorem

Theorem 5.1 establishes ResNet approximation under Setting 4.1, with bounds on approximation error and parameter count. Its proof turns a selected MLP estimator realization into a ResNet and treats ridge-sum data under additional activation and nonlinearity conditions.

Main theorem

  • Under Setting 4.1, Theorem 5.1 provides ResNets indexed by dimension d and accuracy ε, subject to the stated assumptions and cost exponent.The hypotheses include a globally Lipschitz nonlinearity and polynomial growth of the initial data.Source: §5 · p. 16

Special case

  • For ridge-sum data, admissible sigmoidal activations, and globally Lipschitz truncations, part (i)'s conclusions also hold with constants independent of d, ε, and the direction vectors.The theorem states that for every ξ > 0, the networks in this case may be chosen with a bound involving Cξ.Source: §5 · p. 16

Proof strategy

  • The proof represents a selected MLP estimator sample exactly as a ResNet, preserving the sample whose error was established in the preceding estimate.The mean-square error estimate is applied to the estimator built from surrogate data before the representation step.Source: §5 · pp. 17–18
  • Lemma 4.10 and the preceding estimates are combined to bound the parameter count of the resulting ResNet.For part (i), the proof fixes ξ = 1 and chooses η to establish the parameter estimate.Source: §5 · p. 18

6 Consequences and examples

The examples apply the theorem to a truncated Allen–Cahn reaction and describe an extension from Lebesgue measure to certain probability measures. The Allen–Cahn truncation agrees with the classical reaction on the solution range.

Allen–Cahn-type example

  • The truncated reaction agrees with the classical Allen–Cahn reaction on the solution range [-1, 1], and its truncation level is independent of dimension.The comparison principle keeps the solution in this interval because the initial data lie in [-1, 1] and ±1 are solutions.Source: §6 · p. 19
  • The Allen–Cahn example therefore inherits the bound (46) for every ξ > 0.This follows by applying Theorem 5.1(ii) to the globally Lipschitz truncation.Source: §6 · p. 19

General measures

  • For probability measures νd with the stated moment bound, the theorem's Lebesgue-measure error can be replaced by an L2(νd)-error.The explicit exponents from part (ii) persist when γ ≤ 2.Source: §6 · p. 19

7 Conclusion

The paper proves a deterministic ResNet expressivity result for semilinear heat equations under network-surrogate assumptions on the data. Its accumulator construction represents an MLP-estimator realization, while broader PDE settings and constructive training remain open.

Conclusion

  • For admissible sigmoidal activations, globally Lipschitz truncations, and ridge-sum data, the result gives a parameter bound Cξd^(4+ξ)ε^-(3+ξ) for every ξ > 0.More generally, the result applies to semilinear heat equations whose data admit the network surrogates of Setting 4.1.Source: §7 · p. 19
  • The accumulator construction represents one MLP-estimator realization by concatenating residual updates, so parameter counts add without requiring an identity representation of the activation.Source: §7 · p. 19
  • Extensions to state-dependent coefficients, gradient-dependent nonlinearities, and space–time approximations, along with sharper exponents and constructive training, remain open.Source: §7 · p. 19

A Proofs of the quantitative MLP inputs

The appendix supplies quantitative inputs used by the main theorem, including growth and perturbation estimates for the solution and a complexity estimate for the MLP level. It also identifies the cited convergence result used to control the full estimator error.

Proof of Lemma 3.2

  • A backward Grönwall argument bounds the solution's growth using the initial-data bound and Gaussian moments of Brownian motion.The proof first bounds the solution at random points x + Wt and applies the inequality over times in [0,T].Source: A Proofs of the quantitative MLP inputs · p. 20

Proof of Lemma 3.3

Proof of Corollary 3.4

  • The full-history recursive MLP estimator's full L2-error is controlled using the cited MLP convergence estimate under the Lipschitz and qth-moment hypotheses.The text identifies [37, Theorem 3.5], in the form of [36, Corollary 2.4].Source: A Proofs of the quantitative MLP inputs · p. 21

Proof of Lemma 3.5

  • Lemma 3.5 chooses the least level N meeting the error target and bounds its associated cost using ξ-dependent constants.The argument uses the decay of EN and handles the fixed initial levels by absorbing their cost into a constant.Source: A Proofs of the quantitative MLP inputs · p. 22

B Proof of the one-dimensional approximation lemma

The proof builds sigmoid-based staircases to approximate Lipschitz and finite-total-variation functions. It realizes the approximants as feedforward networks, with a variation-adapted construction giving a uniform error of δ and width at most ⌈3T/δ⌉−1.

Step 1: The staircase target

  • For a grid on [−R,R] with spacing 2R/n, the proof constructs a midpoint staircase from sampled values of ϕ.The staircase is constant outside its central region; if ϕ is constant on the two exterior intervals, the same estimate extends from [−R,R] to R.Source: B Proof of the one-dimensional… · pp. 22–23

Steps 2–3: Sigmoidal smoothing and realization

  • The sigmoidal construction realizes Q_n,R as a network with architecture (1,n,1) and 3n+1 parameters.The proof describes Steps 1–3 as a classical sigmoidal staircase argument using bandwidth equal to the grid spacing.Source: B Proof of the one-dimensional… · pp. 22–23

Step 4: The finite-total-variation assertion

  • The finite-variation construction samples ϕ across cells of variation at most λ, so the resulting staircase differs from ϕ by at most λ before sigmoid smoothing.The increments have magnitude at most λ, and their total absolute sum is at most T.Source: B Proof of the one-dimensional… · p. 24
  • For finite-total-variation ϕ, the variation-adapted staircase and its network approximation achieve sup-norm error at most δ with width at most ⌈3T/δ⌉−1.The construction uses K−1 centers, with cellwise variation at most λ=δ/3 and centers separated by at least λ/L.Source: B Proof of the one-dimensional… · p. 24

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors disclose using ChatGPT and Claude during manuscript preparation and state that they reviewed suggestions, verified the work, and retain responsibility for its content.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

  • ChatGPT and Claude were used to improve manuscript language and readability and to check notation, cross-references, bibliographic details, and mathematical exposition.The tools were also used to flag possible issues in the exposition.Source: Declaration of generative AI and… · p. 25
  • The authors critically reviewed all suggestions and proposed revisions before deciding whether to use them.Source: Declaration of generative AI and… · p. 25
  • The authors independently verified the mathematical statements, proofs, and references, made the final decisions, and take full responsibility for the published article.Source: Declaration of generative AI and… · p. 25
Loading 2609.03626v1…