PaperlayerView exact revision on arXiv

Source-linked AI summary

Size Lowerbounds for Deep Operator Networks

Anirbit Mukherjee, Amartya Roy

arXiv:2308.06338v3cs.LGcs.CCcs.NAmath.APmath.NA

TL;DR

DeepONets offer a way to learn mappings that solve families of PDEs, but the size needed for low training error on noisy data is not well understood. The paper proves a data-dependent lower bound and reports experiments suggesting that increasing the shared branch–trunk output dimension while lowering training error may require training data to grow quadratically with it.

  • Problem

    The paper asks what DeepONet size is needed for low empirical error on noisy data, a relationship between architecture size and training-data availability that prior bounds did not establish.

    Source: §1 · p. 3

  • Method

    The authors prove a data-dependent lower bound for bounded-weight DeepONets and conduct experiments on an advection-diffusion-reaction PDE.

    Source: §1 · p. 3, §7 · p. 16

  • Results

    The common branch–trunk output dimension q must be Ω(n^1/4) for low empirical error, and experiments suggest that lowering training error as q increases may require data to scale quadratically with q.

    Source: §1 · p. 3, §8 · p. 17

  • Takeaways & Limitations

    The results connect DeepONet output dimension to training-data size, with experiments indicating a non-trivial range where empirical risk improves as q increases when data scales quadratically with q.

    Source: §8 · p. 17

    The stated lower bound is a necessary, not sufficient, condition; its presented weakened form assumes a common parameter-norm bound and sigmoid output gates.

    Source: §4 · p. 7

Abstract

from arXiv · show

Deep Operator Networks are an increasingly popular paradigm for solving regression in infinite dimensions and hence solve families of PDEs in one shot. In this work, we aim to establish a first-of-its-kind data-dependent lowerbound on the size of DeepONets required for them to be able to reduce empirical error on noisy data. In particular, we show that for low training errors to be obtained on $n$ data points it is necessary that the common output dimension of the branch and the trunk net be scaling as $\Omega \left ( \sqrt[\leftroot{-1}\uproot{-1}4]{n} \right )$. This inspires our experiments with DeepONets solving the advection-diffusion-reaction PDE, where we demonstrate the possibility that at a fixed model size, to leverage increase in this common output dimension and get monotonic lowering of training error, the size of the training data might necessarily need to scale at least quadratically with it.

1 Introduction

DeepONets learn operators that map inputs to solutions, offering a way to solve families of PDEs without rerunning conventional simulations for each parameter choice. This paper studies the size needed for low empirical error and introduces a data-dependent lower bound for DeepONets, whose branch and trunk networks share an output dimension.

1 Introduction

  • For parametric PDEs, conventional methods such as finite element methods require separate simulations for each parameter choice, while operator learning seeks faster mappings from parameters to solutions.The paper places operator learning within the broader physics-informed machine-learning program.Source: §1 · p. 1

1.1 The Formal Setup of DeepONets

  • DeepONets use paired branch and trunk networks to learn operators between function spaces, making them a way to solve a family of parametric PDEs in one shot.The branch takes discretized function values at sensor points; the trunk takes a point in the solution domain, and both produce q-dimensional outputs.Source: §1 · p. 2, §1.1 · p. 3

1 Introduction

  • The paper targets a gap in explaining overparameterization: prior size-independent generalization bounds do not explain why larger models may be needed for good performance.The authors note that the Lipschitz-based argument of Bubeck & Sellke (2023) does not fit DeepONets, which are not Lipschitz functions.Source: §1 · pp. 2–3
  • The common output dimension q must be Ω(n^1/4) for a bounded-weight DeepONet class to achieve empirical error below a label-noise-dependent threshold with high probability.The suppressed prefactors scale inversely with the weight bound and model size.Source: §1 · p. 3
  • At fixed model size, experiments are intended to test whether increasing q while lowering training error may require training data to scale at least quadratically with q.The paper presents this as a possibility to demonstrate experimentally, not as a general established scaling law.Source: §1 · p. 3

2 Related Works

Earlier DeepONet lower bounds do not establish how required architecture size relates to available training data. The paper distinguishes its result by targeting noisy supervised data and a PDE-independent setting.

2 Related Works

  • The earlier DeepONet lower bound considered the Burgers PDE and does not establish a relationship between required architecture size and available training data.The cited setup concerns the operator mapping initial data to the solution at time π/2.Source: §2 · p. 5
  • This paper’s lower bound targets supervised training with noisy labels, asking when empirical error can fall below a measure of label noise.The earlier bound instead assumes no noise in training labels.Source: §2 · p. 5
  • Unlike the Burgers-specific prior bound, the paper’s theorem is PDE-independent.The authors identify connecting architecture size to available training data as a key distinction.Source: §2 · p. 5

3 Our Setup

The setup defines bounded branch and trunk function classes and builds DeepONets by pairing their outputs. It also illustrates the training data and empirical loss with a forced-pendulum example.

3 Our Setup

  • The setup models each training example as an input-output pair sampled i.i.d. from a distribution, with g(s,p) defined as the conditional expectation of y given (s,p).The labels are bounded in [−B,B].Source: §3 · p. 5

Definition 2. Branch Functions & Trunk Functions

  • Branch and trunk functions map their respective inputs to R^q and are constrained by parameter counts, Lipschitz bounds, weight norms, and output bounds.The output bound abstracts networks whose final layer uses bounded activations, while earlier layers may use activations such as ReLU.Source: Definition 2. Branch Functions & Trunk… · pp. 5–6

Definition 3. DeepONets

  • The DeepONet class H consists of functions formed by taking the inner product of a branch function and a trunk function.The setup also assumes a weight-distance inequality for valid branch and trunk weights; the authors note that this implicitly constrains the data-distribution support.Source: Definition 3. DeepONets · p. 6
  • For the forced-pendulum example, training examples pair a discretized forcing function, a time instant, and the pendulum’s angular position; the empirical loss is squared error.The forcing enters the branch net and time enters the trunk net.Source: Definition 3. DeepONets · pp. 6–7

4 The Main Theorem

The main theorem gives a necessary lower bound on DeepONet output dimension for achieving low empirical error on sampled noisy data, under bounded-weight and parameter assumptions. Experiments report a related training-data scaling pattern that goes beyond the theorem’s guarantee.

4 The Main Theorem

  • The bound is necessary, not sufficient; the sigmoid-gate specialization assumes common weight and total-parameter bounds.The theorem’s interpretation also assumes the branch net has an α-fraction of the parameters.Source: §4 · p. 7
  • q=Ω(n^1/4) is necessary for a predictor to achieve empirical error below a label-noise-dependent threshold with high probability under the theorem’s assumptions.The result concerns the common output dimension of the branch and trunk nets.Source: §4 · pp. 7–8
  • In experiments, performance improved monotonically as training-data size n and output dimension q increased with q√n nearly constant, while a slightly smaller growth rate for n broke monotonicity.These experiments used different DeepONet architectures with nearly the same parameter count on a fixed PDE; the authors note this scaling law is outside the theorem’s analysis.Source: §4 · p. 8

5 Lemmas Towards Proving Theorem 4.1

The proof develops covering and concentration lemmas to relate DeepONet empirical error to label noise, culminating in a connection between low empirical error and output–noise correlation.

5 Lemmas Towards Proving Theorem 4.1

  • Lemma 5.1 bounds the covering number of the joint weight space using covers of the branch and trunk weight spaces.The proof constructs separate covers and shows their Cartesian product covers the joint space.Source: §5 · p. 8, §5.1.1 · p. 9

5.1.2 Proof of Lemma 5.2

  • Lemma 5.2 quantifies how empirical risk changes when branch and trunk weights are replaced by nearby weights.The proof approximates each weight vector within θ/2 and bounds the resulting risk change using the DeepONet inner product and Cauchy–Schwarz.Source: §5 · p. 8, §5.1.2 · pp. 9–10

5.1.3 Proof of Lemma 5.3

  • Lemma 5.3 bounds deviations involving label noise for predictors in a cover of the DeepONet class.The proof uses concentration for bounded random variables and a union bound over covered predictors.Source: §5 · p. 8, §5.1.3 · pp. 10–13

5.1.4 Proof of Lemma 5.4

  • Lemma 5.4 links low empirical error to correlation between a DeepONet’s output and label noise.Its proof combines events controlling noise norms, noise–signal inner products, and the existence of a low-risk predictor.Source: §5 · p. 9, §5.1.4 · p. 14

5.1 Proofs of the Lemmas

  • The section supplies proofs of the intermediate lemmas used to establish Theorem 4.1.The proofs use Euclidean covering arguments, risk perturbation bounds, and concentration inequalities.Source: §5.1 · p. 9, §5.1.1 · p. 9, §5.1.2 · p. 10

6 Proof of the (Main)Theorem 4.1

The proof transfers a low-risk predictor to a finite cover of the DeepONet class, then uses probability bounds to derive a necessary lower bound on q.

6 Proof of the (Main)Theorem 4.1

  • A concentration bound for the covered class yields a necessary condition for the required probability of finding a low-risk predictor.The proof combines probability bounds and weakens the resulting inequality to obtain the lower bound.Source: §6 · pp. 14–16
  • The proof applies the risk-perturbation lemma to replace a predictor with one in an ε/(2q²)-cover while bounding the increase in empirical risk.It then defines events for the original class and the cover, transferring existence of a sufficiently low-risk predictor to the cover.Source: §6 · p. 15
  • The proof concludes that a necessary lower bound on q follows from the derived inequality and q ≤ min{dB,dT}.The argument weakens the preceding necessary condition to obtain the theorem’s stated bound.Source: §6 · p. 16

7 The Experiment Set-up

The experiments use DeepONets to learn the source-to-solution operator for an advection-diffusion-reaction PDE, varying output dimension and training-data size at approximately fixed model size. Their results show that increasing q can lower training loss when data grows sufficiently, while a slower data increase breaks that monotonic trend.

7 The Experiment Set-up

  • The study uses the advection-diffusion-reaction PDE and trains a DeepONet to map source terms f(x) to solutions u(x,t).For preliminary experiments, the diffusion coefficient and reaction rate are both 0.01; reference solutions are obtained with a second-order finite-difference method.Source: §7 · p. 16
  • Training data are generated from a mean-zero Gaussian random field with an RBF covariance kernel and length scale 10^-3.The PDE is solved for sampled inputs to obtain reference solutions used in training.Source: §7 · p. 16

7.1 Implementations & Results

  • The experiment targets a fixed total parameter count while varying the common output dimension and training-data size.Each setting uses ten depth-5 models with widths from 24 to 50; branch and trunk inputs have dimensions 40 and 2, respectively.Source: §7.1 · p. 17
  • With q√n approximately fixed, increasing q and n produces an almost monotonic improvement in performance.Figure 2 plots training-loss dynamics for ten models over 120 epochs; q varies from 5 to 50 in increments of 5, with starting n = 10^4.Source: §7.1 · p. 17, Figure 2 · p. 17
  • When the data-size increase is insufficient, the earlier monotonic improvement with increasing output dimension breaks down.The authors attribute the difference between the two experiments to the rate of increase in training data.Source: §7.1 · p. 17, Figure 3 · p. 17

8 Discussion

The paper presents a data-dependent lower bound on the common output dimension q and experiments suggesting that, at fixed model size, increasing q can improve empirical risk when training data scales quadratically with q. The proof does not yet exploit the branch and trunk network structure, and the bound is agnostic to the PDE being solved.

8 Discussion

  • The experiments suggest a non-trivial range of q where empirical risk improves at fixed model size if training data scales quadratically with q.The authors identify proving this observed scaling law as a direction for future theoretical work.Source: §8 · p. 17
  • The theoretical result does not yet fully exploit branch and trunk network structure and is currently agnostic to the PDE being solved.The authors suggest that extending the proof could yield bounds that distinguish PDEs by difficulty for operator learning.Source: §8 · p. 18

A Appendix

The appendix varies the common diffusion and reaction values and finds that the scaling-law pattern persists across those settings. At fixed q√n, best empirical loss falls monotonically as q increases; in the fixed q 2 3 setting, this monotonicity breaks, particularly for higher D = k.

A Appendix

  • The experiments extend the earlier demonstration by revisiting it at diffusion and reaction values spanning orders of magnitude above and below D = k from Section 7.1.Source: A Appendix · p. 22
  • For each tested D = k pair, best empirical loss falls monotonically with increasing q at fixed q√n, while monotonicity breaks at fixed q 2 3, particularly for higher D = k.The four pairs are 1, 0.1, 0.001, and 0.0001.Source: A Appendix · p. 22, Figure 4 · p. 22, Figure 5 · p. 22
  • The authors conclude that the possible scaling law for DeepONets continues to hold across these experiments.Source: A Appendix · p. 22

A.1 Experiment at a fixed

This appendix experiment uses increasing q and n settings inspired by Theorem 4.2, with nearly fixed net size and q 1 6. Its plots report training loss versus epoch, and the authors say monotonic performance improvement with increasing q continues at this scaling.

A.1 Experiment at a fixed

  • The experiment chooses increasing q and n settings inspired by Theorem 4.2, with q 1 6 and operator-net size kept nearly constant.The experiments use common D and k values of 0.0001 and 1.Source: §A.1 · p. 24
  • Figure 8 plots training loss versus epoch for the fixed q setting.Source: Figure 8 · p. 24, §A.1 · p. 24
  • Monotonic performance improvement with increasing q continues at the scaling used in these experiments.Source: §A.1 · p. 24

A.2.1 Experiments where

The appendix tables identify which columns correspond to figures from the experiments. Their captions specify mappings but do not report the table values.

A.2.1 Experiments where

A.2.2 Experiments where

  • Table 2’s D = k = 10^-2 column corresponds to Figure 2, and its last five columns correspond to the left columns of Figures 4–7.Source: Table 2 · p. 24

A.2.3 Experiments where

  • Table 3’s D = k = 10^-2 column corresponds to Figure 3, and its last five columns correspond to the right columns of Figures 4–7.Source: Table 3 · p. 25

Related papers

Loading 2308.06338v3…