Source-linked AI summary
Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
Weinan E, Chao Ma, Stephan Wojtowytsch, Lei Wu
TL;DR
Neural-network machine learning is powerful yet fragile, and its mathematical foundations remain incomplete. This review synthesizes rigorous results, numerical experiments, and simplified models to explain approximation, generalization, optimization, and training dynamics while identifying open problems and scope limits.
Problem
The paper addresses why neural networks succeed despite fragility and which mathematical questions about their approximation, loss landscapes, and training remain unresolved.
Method
The review combines rigorous mathematical results with careful numerical experiments and simplified-model analysis across neural-network function spaces, optimization, and training dynamics.
Results
The review identifies a broad emerging picture, including natural function-space measures for several architectures, while highlighting major unresolved questions about loss landscapes and training.
Takeaways & Limitations
Mathematical understanding has advanced across approximation and training, but the review presents these results as progress toward, rather than completion of, a unified account.
Takeaways & Limitations
The Barron space may be too large for studying trainability because some functions with polynomial Barron norms have exponentially small gradients.
Abstract
from arXiv · showhide
The purpose of this article is to review the achievements made in the last few years towards the understanding of the reasons behind the success and subtleties of neural network-based machine learning. In the tradition of good old applied mathematics, we will not only give attention to rigorous mathematical results, but also the insight we have gained from careful numerical experiments as well as the analysis of simplified models. Along the way, we also list the open problems which we believe to be the most important topics for further study. This is not a complete overview over this quickly moving field, but we hope to provide a perspective which may be helpful especially to new researchers in the area.
1 Introduction
The introduction frames neural-network machine learning as powerful but fragile, and sets mathematical understanding, robust models, and unresolved optimization and generalization questions as central goals.
- Neural networks can approximate high-dimensional functions efficiently and accurately, but their success depends on numerous tricks and difficult parameter tuning.
- The review aims to explain neural-network success and subtleties while developing models that are equally successful but less fragile.
- The article combines rigorous mathematics with numerical experiments and simplified-model analysis to review achievements and identify remaining puzzles.
- Continuous formulations are attractive because they recover scaled versions of existing models and may improve robustness to hyper-parameter choices, but their superiority remains unestablished.
- The review treats supervised learning as high-dimensional function approximation, separating approximation error from estimation error and emphasizing generalization beyond training data.
- It highlights unresolved questions about hypothesis spaces, non-convex loss landscapes, training algorithms, and the behavior of over-parametrized models.
2 Preliminary remarks
The preliminary remarks position neural networks within approximation theory and survey their expressive power, dimensionality challenges, optimization behavior, and the review’s deliberate scope.
- Universal approximation guarantees uniform approximation of continuous functions on compact domains, but offers limited quantitative information and does not by itself resolve high-dimensional complexity.
- Barron’s estimate provides a convergence rate independent of dimension, although its constant may still depend on dimension through Fourier regularity.
- Over-parametrized networks can fit arbitrary noisy data, yielding many global minima, while selecting solutions that generalize well may depend on implicit regularization from training dynamics.
- The review adopts a numerical-analysis perspective and focuses on noiseless L2 regression, omitting classification, several architectures, realistic stochasticity, training tricks, and data-dependent geometry.
3 The approximation property and the Rademacher complexity of the hypothesis space
This section develops approximation and generalization viewpoints for neural-network hypothesis spaces, emphasizing function-space structure, quantitative error bounds, and random-feature models.
- Neural-network approximation theory seeks natural function spaces supporting both direct approximation results and inverse characterizations of efficiently approximable functions.
- Generalization is studied through the complexity of bounded function classes, such as Rademacher complexity, alongside approximation error.
- A single norm may not control approximation and estimation errors simultaneously, motivating generalized function-space measures.
- 3.1 Random feature model: Random-feature models are naturally associated with the reproducing kernel Hilbert space induced by their feature kernel.
- 3.1 Random feature model: The direct and inverse approximation theorems characterize when target functions admit efficient random-feature representations and what this implies about their function-space membership.
- Regularized random-feature estimators admit population-risk bounds with high probability, although sharper rates require explicit eigenvalue-decay assumptions.
3.2 Two-layer neural network model
Two-layer networks are naturally analyzed through Barron space, which characterizes representable functions and supports direct and inverse approximation results. The section also relates Barron complexity to generalization and distinguishes the Barron norm used here from a spectral norm.
- Barron space: Barron space provides the function class naturally associated with two-layer neural networks, including finite neuron sums and functions such as the Euclidean norm.For ReLU networks, the Barron norm is defined through a probability distribution over neuron parameters.
- Norms and assumptions: The Barron norm used in this section differs from the Fourier-transform-based spectral norm used in earlier work.The distinction matters because the two norms quantify complexity through different constructions.
- Barron space: Every Sobolev function in Hs(Rd) with s > d is Barron, while Barron functions are Lipschitz and exclude distance functions to curved surfaces.The structure theorem decomposes Barron functions into components smooth away from affine subspaces.
- Approximation theorems: For any Barron function and neuron count m, two-layer networks achieve direct approximation bounds in both L2 and L∞ settings.The section also gives a matching inverse approximation theorem: uniformly convergent networks with controlled complexity imply membership in Barron space.
- Generalization: The Rademacher complexity of Barron-norm balls has a Monte Carlo-like rate, and regularized estimators admit high-probability population-risk bounds under suitable norm control.These results connect the Barron representation to statistical generalization rather than approximation alone.
3.3 Residual networks
Residual networks are modeled through flow-induced function spaces arising from an ODE limit of residual architectures. These spaces support direct and inverse approximation results, but available generalization theory remains restricted and some representation choices are unexplored.
- Limitations: The ODE model can express no more functions with large m than with m = 1, although m may change the associated norm and approximation constant.The effect of choosing m ≠ 1 is explicitly left unexplored.
- Approximation theorems: Residual networks admit direct approximation for functions in ˜D2 and inverse approximation under convergence and bounded-parameter assumptions.The inverse result places the limiting function in D∞ and provides additional bounds when D1 norms remain uniformly controlled.
- Generalization: Weighted path norms provide the available route to Rademacher-complexity and generalization bounds for residual networks.The analysis uses a modified flow-induced norm for complexity and a discrete weighted path norm for empirical-risk regularization.
3.4 Multi-layer networks: Tree-like function spaces
Tree-like function spaces organize multilayer networks by depth and path-based complexity, extending the one-hidden-layer Barron perspective. They support approximation, compactness, inverse results, and generalization bounds, but their approximation rate remains depth-dependent.
- Approximation theory: The closed unit balls are compact, and bounded-complexity convergent network sequences satisfy an inverse approximation theorem at the same depth.Compactness holds in C0(K) for compact K and in L2(P) for compactly supported P.
- Approximation and generalization: Tree-like networks have direct approximation and generalization bounds based on path complexity, with Rademacher dependence that can improve from 2L to L3 under stronger norm control.The standard direct approximation result does not have a Monte Carlo rate.
- Approximation and generalization: A finite multilayer network construction achieves the stated L2 approximation bound with error 2L ∥f∗∥WL / √m.The construction uses layer widths satisfying mℓ = mL−ℓ+1 and has O(m2L−1) weights before tree-like rearrangement.
- Limitations: It remains unclear whether multilayer networks can achieve an approximation rate independent of depth.The section attributes the depth dependence to the different conditional-expectation structure arising when multilayer networks are discretized.
- Tree-like spaces: Tree-like spaces form a depth scale, are stable under composition, and coincide with Barron space at depth one.A function at depth ℓ remains in the space at any larger depth L, while composing depths L and ℓ yields depth L+ℓ.
3.5 Indexed representation and multi-layer spaces
Indexed representations model arbitrarily wide multilayer networks as functions over probability spaces and provide complete metric spaces for their analysis. These spaces relate to tree-like and Barron spaces, but their exact agreement remains unresolved for sufficiently expressive multilayer models.
- Indexed representations: Indexed representations define arbitrarily wide networks through measurable weight functions on layer-specific probability spaces, recovering finite networks when the spaces are finite.With sufficiently expressive index spaces, the resulting class can form a vector space because networks can be decomposed and added.
- Relation to function spaces: Arbitrarily wide networks form a finite-path-norm subspace of the depth-L tree-like space, and one-hidden-layer networks coincide with Barron space when the index space is (0, 1).For multiple hidden layers, the arbitrarily wide class may be a proper subspace because non-consecutive layers do not share an index.
- Open question: For two hidden layers, the indexed class is a separable Barron subspace, while the broader tree-like and indexed spaces are not yet known to coincide.The indexed class nevertheless contains Barron functions and their compositions, including products of Barron functions.
- Training dynamics: Restricting measurable weight functions to L2 weights is motivated by the fact that L2 weight control controls the path norm and supports a natural gradient-flow structure.The proof discussed is specific to network-like architectures and does not generalize to tree-like structures.
- Metric structure: The resulting multi-layer spaces are complete metric spaces, with normalization required to prevent equivalent layer rescalings from having zero distance.The metric is designed to respect the parameter structure of the network.
3.6 Depth separation in multi-layer networks
Depth can substantially increase representational power, but the systematic separation between successive depths remains unresolved. The review also emphasizes that low-complexity spaces may approximate some structured functions well while poorly approximating general high-dimensional classes.
- Depth separation in multi-layer networks: Two hidden layers can exactly represent compositions of two Barron functions, whereas one-hidden-layer approximation may require substantial complexity.The criterion is practically checkable, but exact representability does not establish finite-network approximation behavior.
- Depth separation in multi-layer networks: A two-hidden-layer network can be more flexible than a one-hidden-layer network, including through activation constructions that yield dense families of continuous functions.The cited construction uses an analytic, strictly increasing, bounded activation, but is described as impractical.
- Depth separation in multi-layer networks: A composition of Barron functions can have low-complexity components while escaping the approximation behavior of a single Barron-function representation in high dimension.The supporting argument uses Fourier concentration near ridge directions and the small mass of high-dimensional neighborhoods.
- Depth separation in multi-layer networks: A systematic approximation picture for deeper networks is still missing, including known separations between L and L + 1 hidden layers.Existing results show advantages for some significantly deeper networks, but do not yet provide a general theory.
- Tradeoffs between learnability and approximation: Low-complexity function spaces are poor approximators of general Lipschitz classes, even though different spaces vary substantially in their approximation capacity.The review contrasts reproducing kernel Hilbert spaces, Barron space, and tree-like three-layer spaces.
3.8 A priori vs. a posteriori estimates
The review distinguishes a priori estimates, based on the target function, from a posteriori estimates, based on the learned model. Each has important limitations: the former may be unavailable or loose, while the latter can be vacuous or omit approximation error.
- A priori vs. a posteriori estimates: A priori estimates depend on the target function and can show that suitable hypothesis spaces avoid curse-of-dimensionality effects in generalization error.The review regards these rates as potentially near-optimal, apart from multi-layer cases.
- A priori vs. a posteriori estimates: A priori bounds are difficult to quantify because target-function norms are unknown and representation infima are not recovered by a single learned representation.Even exact norm values would not make the bounds tight because Monte Carlo approximation estimates are loose.
- A priori vs. a posteriori estimates: A posteriori estimates depend on the model output and can be evaluated directly, but their norm values are often so large that the bounds are vacuous.Their practical usefulness has not been confirmed in practice.
- A priori vs. a posteriori estimates: A posteriori bounds control only the generalization gap, so strongly constrained spaces can still incur large approximation error.The review notes that this occurs for some norm-based a posteriori estimates.
- Open problems: Sharper approximation and generalization estimates remain open problems, especially beyond Monte Carlo rates for multi-layer spaces.The review also calls for complexity bounds that account for the small pointwise errors in risk integrands.
4 The loss function and the loss landscape
Neural-network loss landscapes are non-convex and can contain complicated critical-point structure, yet several simplified and over-parameterized models have favorable geometric properties. A general mathematical description for large networks remains unavailable.
- The loss function and the loss landscape: As network size increases, the loss landscape is reported to simplify despite arbitrarily bad finite-size examples.This provides a broad qualitative picture, not a complete characterization of optimization dynamics.
- The loss function and the loss landscape: For linear networks, every local minimum is global, while non-global critical points are saddles; bad saddles occur beyond three layers but not at three layers.These results concern linear neural-network models.
- The loss function and the loss landscape: Under various conditions, over-parameterized quadratic networks have only global local minima and strict saddle points.Strict saddles have directions of strictly negative curvature.
- The loss function and the loss landscape: For smooth over-parameterized networks, global minimizers generically form a smooth m − n dimensional submanifold of parameter space.Here m is the number of free parameters and n is the training-data size.
- The loss function and the loss landscape: The curvature of the global-minimizer set helps explain why initialization can lead to different global minimizers.Numerical experiments are cited as observing this lack of positive definiteness.
- Open problems: A general tool is still lacking for determining local minima and their basins of attraction in large neural networks.The corresponding infinite-dimensional limiting landscape is also not clearly formulated.
5 The training process: convergence and implicit regularization
The review connects convergence and implicit regularization to model scaling, optimization dynamics, and optimizer stability. Its results show that training behavior can vary sharply across regimes, with limitations from dimensionality, initialization assumptions, and optimizer-specific stability.
- 5.1 Two-layer neural networks with mean-field scaling: Gradient-flow convergence to a global minimizer is established under stated conditions, including smooth initialization and full support of the probability distribution.The result concerns the training algorithm and uses PDE arguments; convergence in Wasserstein metric is additionally required for the limiting distribution to be a minimizer.
- 5.1 Two-layer neural networks with mean-field scaling: High-dimensional targets outside Barron space may cause a dynamic curse of dimensionality, making gradient-descent training difficult despite convergence results for Barron functions.The review contrasts apparently dimension-independent convergence for Barron functions with difficulty approximating low-Barron-norm representations in high dimensions.
- 5.2 Two-layer neural networks with conventional scaling: Highly over-parametrized networks can converge exponentially to global minima, but their generalization is no better than the associated random feature model and lacks implicit regularization.The review describes this regime as theoretically convergent but practically disappointing.
- 5.2 Two-layer neural networks with conventional scaling: Conventional scaling produces distinct training regimes: neural-network dynamics can improve beyond random features, whereas random-feature-like behavior can sharply worsen generalization.The neural-network phase includes quenching of most neurons and activation of a few; the review links regime changes to test-error phase transitions.
- 5.4 Double descent and slow deterioration for the random feature model: For random-feature models, test error peaks near m = n when the Gram matrix’s smallest eigenvalue is very small, but the resonance may be hidden until extremely long training.The reported dynamics can show an initial error decrease, a long low-error interval, and later deterioration as small eigenvalues contribute.
- 5.5 Global minima selection: Optimizer stability selects different minima: SGD favors solutions satisfying a sharpness–non-uniformity bound, while Adam’s behavior changes among spike, oscillation, and divergence regimes.For Adam, small and comparable a and b values are associated with small stable loss, whereas sufficiently larger b produces spikes and instability.
6 Concluding remarks
The concluding remarks synthesize theoretical, numerical, and analytical findings on neural network approximation, training dynamics, and generalization, while identifying open problems requiring rigorous analysis and carefully designed experiments.
- Approximation/generalization properties of hypothesis space: Dimension-independent error rates have been established for regularized models, providing benchmarks for comparing machine learning models and studying implicit regularization.
- Training dynamics for highly over-parametrized neural network models: Highly over-parametrized networks can exhibit exponential convergence of empirical risk, while their generalization is no better than corresponding random feature or kernel methods.
- Mean-field training dynamics for two-layer neural networks: The review highlights mean-field training dynamics for two-layer networks as a major established direction alongside approximation theory and over-parametrized training.
- Concluding results: Established findings also include global-minimum convergence under full-support initialization, interpolation by over-parametrized networks, double descent, adaptive optimization behavior, and phase transitions in generalization.
- Open problems: Open problems include carefully designed experiments comparing network depths and scaled versus unscaled residual networks, while continuous formulations warrant separate study.
A Proofs for Section 3.1
These proofs establish compactness-based subsequence convergence for mean-field measures and derive finite-neuron consequences through probabilistic and complexity arguments.
- Compactness argument: Tightness of the probability measures yields a weakly convergent subsequence through Prokhorov’s theorem.
- Limit identification: The limiting representation uses conditional expectations and bounded continuous test functions to pass from finite-neuron measures to the limiting measure.
- Finite-sample estimate: The proof proceeds from a proposition giving a high-probability finite-sample statement for randomly sampled weights.
- Complexity bound: A regularized empirical-risk comparison bounds the solution norm, controls the associated hypothesis class’s Rademacher complexity, and completes the proof.
B Proofs for Section 3.2
The proof uses ReLU homogeneity, random signs, contraction, and norm duality to construct weights satisfying the required complexity inequality.
- ReLU homogeneity permits normalization of the weights and identification of the coefficient magnitude with the Barron norm.
- Independent Rademacher signs remove coefficient signs and enable contraction-based bounds for the relevant complexity expression.
- The argument invokes duality between ℓ1 and ℓ∞ norms and Hilbert-space Rademacher complexity to obtain the required estimate.
- The resulting inequality guarantees the existence of a set of weights satisfying the stated bound.