PaperlayerView exact revision on arXiv

Source-linked AI summary

The Modern Mathematics of Deep Learning

Julius Berner, Philipp Grohs, Gitta Kutyniok, Philipp Petersen

arXiv:2105.04026v2cs.LGstat.ML

TL;DR

Many observed deep-learning phenomena lack theoretical explanation. The article defines its scope as gradient-trained deep neural networks and surveys recent mathematical advances aimed at these questions.

  • Problem

    Many practically observed deep-learning phenomena are not theoretically explained.

    Source: §1 · p. 3

  • Method

    The article defines deep learning as gradient-trained deep neural networks and surveys recent mathematical advances.

    Source: §1 · pp. 2–3

  • Results

    The article assembles recent advances aimed at answering these questions.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Theoretical insight can help explain observed phenomena and drive the development of improved methods.

    Source: §1 · p. 3

    The brief introduction cannot cover the full literature and omits many influential architectures, including graph neural networks, autoencoders, and generative adversarial networks.

    Source: §6 · p. 44

Abstract

from arXiv · show

We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail.

1 Introduction

This section introduces the emerging mathematical analysis of deep learning as a response to phenomena and questions classical learning theory does not explain, including optimization, approximation, generalization, and what neural networks can fit. It also frames the need to understand deep learning’s successes in natural-science problems, learned features, and effects of architecture.

1 Introduction

  • Deep learning has succeeded across applications, including a protein-folding predictor matching lab-based accuracy and transformer models used for text generation and symbolic mathematics.The passages also describe advances in games, image classification, drug development, molecular dynamics, and high-energy physics.Source: §1 · p. 2
  • The article defines deep learning narrowly as training deep neural networks with gradient-based methods, then offers an intuitive, selective survey rather than a complete overview.Its two ingredients have historical roots, but the field's questions remain inadequately explained by classical learning theory.Source: §1 · pp. 2–4
  • Mathematical theory can explain practical phenomena, guide improved methods, and support control, predictability, fairness, and explainability in high-stakes applications.The article also presents theoretical study as motivated by scientific curiosity.Source: §1 · p. 3
  • The field studies why deep networks generalize despite overparameterization, benefit from depth, avoid dimensionality barriers, optimize successfully, learn features, and excel in scientific applications.It also asks which architectural aspects affect performance and how.Source: §1 · p. 3

1.1 Notation

  • The notation subsection establishes conventions for mathematical objects, norms, function spaces, and probability used in the article.Source: §1.1 · p. 4

1.2 Foundations of learning theory

  • The foundations subsection formulates learning as selecting a hypothesis set and algorithm to fit training data and generalize, then introduces neural-network hypotheses and empirical risk minimization.It notes that classical generalization analysis typically assumes training and unseen data are independent samples from the same distribution.Source: §1.2 · pp. 4, 5, 8
  • No learner avoids distributions with arbitrarily slow expected-risk convergence [DGL96, Theorem 7.2]; useful bounds require prior regularity, as larger hypothesis classes reduce approximation error but impede generalization.The approximation–generalization tension motivates seeking a suitable hypothesis-class complexity.Source: §1.2 · pp. 9–10

1.2.1 Optimization

  • Gradient-based methods, especially SGD with backpropagated derivatives, are commonly used to approximately minimize empirical risk, but nonconvex objectives need not yield global minima.SGD can converge to a local, non-global minimum; stochastic fluctuations may also help escape saddle points under various conditions.Source: §1.2.1 · pp. 10–12

1.2.2 Approximation

  • Two-layer networks can uniformly approximate continuous functions on compact sets under activation conditions; the stated criterion is that the activation is not almost everywhere a polynomial.The approximation is in the topology induced by the L∞(K)-norm.Source: §1.2.2 · p. 12
  • With discriminatory activations, two-layer networks with m hidden neurons interpolate arbitrary m labeled points using O(m(d + k)) parameters; nearly all activations permit a three-layer O(d + mk) construction.The two-layer result assumes distinct inputs and a continuous, non-polynomial activation.Source: §1.2.2 · pp. 13–14
  • For smooth targets, networks attain optimal classical approximation rates [Mha96, Theorem 2.1], using activation-function difference quotients to reproduce polynomial approximation, with unbounded parameter magnitudes allowed.The resulting rates explain how the approximation error can decrease as network size grows.Source: §1.2.2 · p. 14

1.2.3 Generalization

  • Under i.i.d. data, bounding uniform generalization error begins by applying concentration inequalities to empirical-risk deviations for each hypothesis.The losses for a fixed hypothesis are i.i.d. under the stated data assumption.Source: §1.2 · p. 8, §1.2.3 · p. 15
  • For bounded losses, Hoeffding’s inequality and a union bound give high-probability guarantees for countable, weighted hypothesis sets, but the bound grows with set size.The weighting must be chosen before seeing the training data; for neural networks, the resulting hypothesis-set sizes can be enormous.Source: §1.2.3 · p. 15
  • For piecewise polynomial activations, neural-network VC dimension scales, up to logarithmic factors, linearly with parameter count and quadratically with depth, supporting finite-VC generalization bounds.The section summarizes this as a high-probability bound for fixed-depth neural-network classification hypotheses.Source: §1.2.3 · pp. 15–16

1.3 Do we need a new theory?

  • Overparameterized neural networks can interpolate training data yet generalize well, even when parameter-count-based VC bounds are vacuous, creating a mismatch with classical bias-variance expectations.The cited CIFAR-10 example trained a ReLU convolutional network with about 1.6 million parameters to zero empirical risk on 50,000 images while achieving close to state-of-the-art generalization.Source: §1.3 · p. 17
  • The puzzle is sharpened by zero training error on random labels, mild test-error sensitivity to explicit regularization, and test-error differences across optimization variants and initializations.These observations suggest that the learning algorithm’s range and optimization dynamics may affect generalization.Source: §1.3 · p. 18
  • Low test error can persist when fitting noisy training data to near-zero empirical risk, and further increasing network size can improve test error.Together, these observations challenge the traditional expectation that fitting training data harder eventually harms generalization.Source: §1.3 · p. 18
  • The role of depth remains open: although width-based results establish approximation capabilities, modern networks may have over 1,000 layers, and depth’s quantifiable benefits need explanation.The section also notes that hidden-layer widths and activation choices can substantially affect network capabilities, while architecture selection is often based on practitioner intuition.Source: §1.3 · pp. 19, 21
  • Neural networks perform adequately on high-dimensional tasks despite classical approximation lower bounds that imply prohibitive parameter requirements as dimension increases.For 1-Lipschitz functions on a 784-dimensional unit cube, the cited lower-bound rate would require O(ε^-784) parameters for uniform error ε.Source: §1.3 · p. 19
  • Despite non-convex empirical risk, which can have suboptimal local minima, saddle points, and regions of small gradients, SGD convergence is often observed; explaining when it occurs remains open.The section further identifies the effects of architectural choices beyond width and depth as a question for analysis.Source: §1.3 · pp. 19, 21
  • The authors argue that mathematical theory should guide architecture choice and identify architectures that work for a specific problem versus those yielding suboptimal results.Source: §1.3 · p. 21
  • Analyzing learned features offers data-aware understanding beyond classical theory’s focus on optimization, generalization, and approximation errors, but unknown practical data distributions impede theoretical analysis.Partial knowledge can motivate studying whether trained networks share expected image invariances. A common belief is that neurons act as layerwise feature extractors, partly grounded in experiments and visual-cortex connections [GBC16, Chapter 9.10]; whether trained networks favor biologically inspired sparse multiscale representations is also an open question.Source: §1.3 · p. 22
  • These methods’ broad applicability to problems previously handled by specialized numerical algorithms is surprising, yet their foundational mechanisms and limitations remain poorly understood.Source: §1.3 · p. 22
  • Deep-learning methods have addressed ill-posed inverse problems [AM¨OS19]—including a fourfold MRI scan-time speedup [ZKS+18]—and many PDEs, especially high-dimensional ones where most methods face the curse of dimensionality.Source: §1.3 · p. 22

2 Generalization of large neural networks

This section surveys four research lines addressing neural-network generalization, while noting that many results remain limited to simplified models. It covers kernel behavior, complexity and margin bounds, optimization’s implicit effects, and double descent.

2.1 Kernel regime

  • In the infinite-width limit, neural-network training becomes functional gradient flow in a deterministic neural tangent kernel, and converges to a minimum-kernel-norm interpolator under stated conditions.The conditions include a positive-definite kernel on the input features and a convex, lower-bounded loss.Source: §2.1 · p. 23

2.2 Norm-based bounds and margin theory

  • For piecewise-linear activations, VC dimension scales asymptotically as P(N)L up to logarithmic factors, leaving classical bounds vacuous unless sample size scales at least linearly with parameters.Heavily overparameterized networks used in practice nevertheless seem to generalize outside this regime.Source: §2.2 · p. 24
  • Norm-based bounds avoid direct dependence on parameter count, while margin bounds trade hypothesis-class complexity against the share of training examples classified with margin at least γ.For positively homogeneous activations, bounds can use Frobenius norms; spectral norms can reduce the depth dependence from exponential to linear or remove it.Source: §2.2 · pp. 24–25

2.3 Optimization and implicit regularization

  • Under gradient flow, normalized parameters can implicitly maximize the margin; in deep linear networks, adjacent layers’ singular-space alignment is conserved and supports convergence results under additional assumptions.The maximum-margin convergence result for deep linear networks requires additional assumptions.Source: §2.3 · p. 27

2.4 Limits of classical theory and double descent

  • Interpolating noisy data does not rule out lower risk in the overparameterized regime; in asymptotic settings, benign overfitting is shown for broader cases, including lazy neural-network training.Some of these settings even achieve optimal risk in the overparameterized regime.Source: §2.4 · pp. 28–29
  • For minimum-norm linear regression, benign overfitting is characterized by covariance eigenvalues, and adding input coordinates can either increase or decrease risk depending on their correlations.The risk also depends crucially on the data distribution and chosen algorithm.Source: §2.4 · p. 29

3 The role of depth in the expressivity of neural networks

Depth can compensate for limited width, though universality fails below a critical width; the section examines depth through approximation results and alternative measures of expressivity.

3 The role of depth in the expressivity of neural networks

  • Very narrow networks can remain universal when depth increases arbitrarily, but universality fails below a critical width [HS17].This is given as an example of depth compensating for insufficient width.Source: §3 · p. 29

3.1 Approximation of radial functions

  • Radial functions can be efficiently approximated by three-layer networks, while two-layer approximation may require width exponential in dimension [ES16].On bounded domains, the three-layer construction uses a number of parameters polynomial in d/ε.Source: §3.1 · p. 30
  • Fourier support of shallow networks lies on neuron-direction lines, which expand to tubes after smoothing and can leave high-frequency radial targets poorly approximated.When the union of these tubes is sparse, the construction yields a lower bound on approximation error.Source: §3.1 · pp. 30–31

3.2 Deep ReLU networks

  • Interpolation differences yield scaled sawtooth functions, enabling approximate squaring and, through polarization, multiplication for smooth-function approximation.Composing these primitives can approximate localized Taylor polynomials.Source: §3.2 · p. 31
  • Deep ReLU networks achieve hp-finite-element approximation rates for piecewise analytic functions, with W 1,2 error ε requiring a logarithmic number of parameters.The cited examples include solutions to elliptic boundary and eigenvalue problems with analytic data.Source: §3.2 · p. 32
  • For non-affine C² functions, ReLU approximation exhibits a depth-width trade-off tied to the number of affine pieces generated by the architecture.The number of pieces scales roughly with the product of hidden-layer widths.Source: §3.2 · p. 32

3.3 Alternative notions of expressivity

  • Composing ReLU networks folds the input space, multiplying region counts and allowing deep networks exponentially more affine linear regions [MPCB14].Deep ReLU networks can exhibit significantly more regions than shallow counterparts.Source: §3.3 · pp. 32–33
  • Expected trajectory length grows exponentially with network depth, and trajectory shapes become more complex empirically.Figure 3.4 illustrates trajectories for randomly initialized networks with differing numbers of hidden layers.Source: Figure 3.4 · p. 33, §3.3 · p. 34

4 Deep neural networks overcome the curse of dimensionality

The section presents three routes by which neural networks can avoid the curse of dimensionality: exploiting low-dimensional data structure, sampling-based approximation, and probabilistic representations of PDE solutions. Each route relies on conditions on the data, target function, or PDE, and the PDE approach still leaves important training questions open.

4.1 Manifold assumption

  • When the data distribution lies on a d′-dimensional manifold with d′ ≪ d, neural networks can approximate using a rate depending on d′ rather than ambient dimension d.The construction uses local coordinate charts, localization, classical approximation, and neural-network composition to reduce the problem to the manifold’s intrinsic dimension.Source: §1.2.1 · p. 11, §4.1 · pp. 34–35

4.2 Random sampling

  • For functions with one finite Fourier moment, two-layer ReLU or sigmoidal networks achieve L2 approximation error n^-1/2 with n hidden neurons.The argument uses random sampling and approximation of the sampled component functions by fixed-size networks.Source: §4.2 · p. 36

4.3 PDE assumption

  • The section’s broader motivation is that high-dimensional classical numerical representations can be inadequate for complicated structures such as wave functions with long-range interactions.The cited example concerns differing interaction-energy predictions from two gold-standard methods for large delocalized molecules.Source: §4.3 · p. 36
  • For linear Kolmogorov PDEs, dimension-independent neural-network approximation is preserved when the initial condition and coefficients have that property and suitable technical conditions hold.The route uses stochastic representations and sampling; the initial condition may be exactly represented by a small ReLU network in many computational-finance applications.Source: §4.3 · pp. 37–38
  • Stable neural-network training for PDE solutions at very high accuracy remains challenging, while analysis of SGD computational cost and convergence is open.Very high accuracy is needed in applications such as quantum chemistry.Source: §4.3 · p. 38

5 Optimization of deep neural networks

The section surveys selected explanations for why optimization can work despite the nonconvex neural-network objective. It discusses loss-landscape structure and wide-network lazy training, while noting that the latter can behave like a linear method.

5 Optimization of deep neural networks

  • Despite nonconvexity, saddle points, suboptimal minima, and flat regions, SGD often performs well in applications, motivating partial theoretical explanations.The classical theory described here does not explain this observed performance.Source: §5 · p. 39

5.1 Loss landscape analysis

  • Under a spin-glass model connection, critical points farther from optimal loss are more unstable, so a local minimum is likely close to the global minimum.The assumptions linking spin-glass models and neural networks are acknowledged as unrealistic in practice, although similar critical-point patterns appear in applications.Source: §5.1 · p. 39
  • Nonincreasing-risk paths to the global minimum imply connected loss level sets and rule out non-global minima when such paths exist from every point.The path existence result is stated for empirical risk.Source: §5.1 · pp. 39–40
  • For sufficiently wide, appropriately randomly initialized networks, a path from a random parameter value to the global minimum can stay within O(ε) of the initial risk.The construction connects paths through a linear subspace of network realizations; Figure 5.2 depicts the path construction.Source: Figure 5.2 · p. 40, §5.1 · p. 41

5.2 Lazy training and provable convergence of stochastic gradient descent

  • With high probability for sufficiently large width, the initialization kernel is positive definite and the empirical-risk gradient norm is lower bounded by n/m times the risk.The kernel’s smallest eigenvalue scales linearly with width, supporting the local gradient bound.Source: §5.2 · pp. 42–43
  • For sufficiently small step sizes and sufficient width, gradient descent converges exponentially to arbitrarily small empirical risk while its iterates stay near initialization.In the infinite-width limit, training is practically equivalent to kernel regression, and the described lazy regime essentially acts like a linear method.Source: §5.2 · pp. 43–44

6 Tangible effects of special architectures

This section examines how specialized architectures shape neural-network behavior, including sparse approximations of certain wide networks and sequential computation through recurrent connections, which can make some computations more efficient than feed-forward networks. The analyses use simplifying assumptions and omit several influential designs.

6.1 Convolutional neural networks

  • CNNs encode spatial locality, reuse parameters across translated patches, and produce equivariant representations; universality depends on the convolution and pooling choices.With padding, CNNs can be universal; circular convolutions without pooling instead universally approximate translation-equivariant functions [Yar18b].Source: §6.1 · pp. 45–46

6.2 Residual neural networks

  • Residual connections address deep-network training obstacles: vanishing or exploding gradients and the degradation of training and test accuracy with increasing depth.Viewing residual networks through dynamical systems and optimal control yields a framework in which strong mathematical guarantees can be derived [TvG18] [LLS19].Source: §6.2 · pp. 46–47

6.3 Framelets and U-Nets

  • Deep convolutional framelets provide a frame-theoretic account of encoder-decoder CNNs such as U-nets and explain skip connections as aiding perfect reconstruction.Layer choices can be informed by the rank of input-associated Hankel matrices; singular-vector bases give a sparse but input-dependent representation.Source: §6.3 · pp. 47, 49

6.4 Batch normalization

  • Batch normalization preserves expressivity but can reduce empirical-risk derivatives and increase optimization smoothness, under conditions such as small γ relative to σ_b or moderate batch size [STIM18].The cited analysis also finds a similar effect for second derivatives; the learnable β and γ add two optimization variables [STIM18].Source: §6.4 · pp. 49–50

6.5 Sparse neural networks and pruning

  • Pruning removes selected trained-network parameters, while the lottery ticket hypothesis proposes that a dense randomly initialized network contains a trainable subnetwork matching its test accuracy within the same iteration budget.A related claim is that sufficiently overparameterized networks may contain subnetworks matching the large network's performance already at initialization.Source: §6.5 · pp. 50–51
  • Certain very wide two-layer ReLU networks admit approximations with at least 99% zero parameters and L2(µ) error at most 15||w^(2)||_1√n.Only the output weight matrix needs to change; the argument is inspired by [BK18].Source: §6.5 · pp. 51–52

6.6 Recurrent neural networks

  • Recurrent networks use cyclic computational graphs and time steps, making them suited to sequential data whose inputs determine the internal state for following inputs.That evolving state provides memory, which is desirable in natural language processing.Source: §6.6 · p. 52
  • Backpropagation through time trains recurrent networks by unfolding their structure, but vanishing and exploding gradients make very deep networks hard to train, motivating gated structures such as LSTMs.Source: §6.6 · p. 52
  • A fixed-size recurrent ReLU network can approximate x → x^2 to any desired accuracy, whereas fixed-size feed-forward ReLU networks have positive error lower bounds for non-affine functions.The recurrent construction uses a self-referential definition of the approximant [BF19, Theorem 4.4].Source: §6.6 · p. 52

7 Describing the features a deep neural network learns

The section examines neural-network features through two perspectives: invariance-oriented scattering transforms and hierarchical sparse representations.

7.1 Invariances and the scattering transform

  • Because deformations form an infinite-dimensional group, deformation-invariant descriptors have potential to reduce the dimensionality of practical problems.The scattering transform is presented as a feature descriptor for this setting.Source: §7.1 · p. 53
  • Scattering coefficients are deformation-stable, almost translation-invariant, and energy-preserving in a suitable asymptotic sense.Their deformation stability depends mildly on the deformation and grows essentially linearly with its first and second derivatives.Source: §7.1 · p. 53

7.2 Hierarchical sparse representations

  • Although recovering ML-CSC coefficients is generally NP-hard, layered thresholding can approximately solve the noisy-signal problem, with stable recovery under suitable model hypotheses.The noise is assumed to have small ℓ2-norm.Source: §7.2 · pp. 54–55
  • Sparse-coding theory models CNN kernels as sparsifying dictionaries and the forward pass as layered thresholding for decomposing a noisy signal.The thresholding analogy maps transposed dictionaries to convolutional kernels and thresholds to bias vectors.Source: §7.2 · pp. 54–55

8 Effectiveness in natural sciences

This section surveys deep-learning applications in inverse problems and PDEs, where mathematical guarantees, data constraints, and physical structure shape what methods can do. It emphasizes hybrid inverse solvers and neural architectures tailored to difficult high-dimensional problems.

8 Effectiveness in natural sciences

  • A considerable gap can remain between approximation-theory predictions and neural-network performance in applications.The section surveys applications in inverse problems and PDE models as settings where deep-learning methods have been used successfully.Source: §8 · p. 55

8.1 Deep neural networks meet inverse problems

  • Inverse-problem solvers fall into supervised end-to-end or model-informed hybrids, semi-supervised learned regularizers, and unsupervised deep priors optimized against the forward operator.The best combination of classical methods and deep learning depends on data availability and accuracy, forward-operator properties, and solution requirements.Source: §8.1 · pp. 55–56
  • Supervised inverse solvers can be limited by synthetic training data and ill-posedness, while operator-aware methods help address the latter; semi-supervised regularizers retain convergence or stability guarantees under natural assumptions.Unsupervised methods place regularization in the network architecture, but the deep prior still lacks profound mathematical analysis in full generality.Source: §8.1 · p. 56

8.2 PDE-based models

  • High dimensionality challenges PDE applications including nonlinear Black–Scholes models with default risk, electronic Schrödinger equations, and fully nonlinear, nonsmooth Hamilton–Jacobi–Bellman equations.For realistic nonlinear Black–Scholes models, no general algorithm provably scaling well with dimension is known, except for Multilevel Picard algorithms.Source: §8.2 · pp. 56–57
  • Neural PDE methods include backward-SDE optimization for semilinear parabolic equations [HJE18], Pontryagin-based sampling for HJB equations [NZGK21], Deep Ritz, and Physics Informed Neural Networks.The backward-SDE approach is reported as successful in high dimensions and supports modeling complex financial derivatives with nonlinear effects such as default risk.Source: §8.2 · p. 57
  • Electronic Schrödinger computation requires wavefunctions respecting the Pauli principle and, empirically, architectures that hard-code cusp conditions; its mathematical properties remain largely unexplored.Large-electron systems lack satisfactory solution algorithms, and different gold-standard methods can give very different energy predictions for large delocalized molecules.Source: §8.2 · pp. 57–58
  • A single Slater determinant cannot accurately represent nontrivial wavefunctions because it omits opposite-spin Coulomb interactions, while the number needed for accuracy scales badly with system size.Known approximation results have an n-independent error rate, but their implicit constant depends at least exponentially on n.Source: §8.2 · pp. 58–59
  • Neural architectures incorporating electron interactions, physical invariances, and cusp conditions make accurate ab initio computations possible for previously intractable molecules.The backflow ansatz is more efficient than plain Slater determinants in the number needed for chemical accuracy, and shared parameters across nuclear geometries support potential-energy-surface computation.Source: §8.2 · pp. 58–59

Related papers

Loading 2105.04026v2…