PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning from a Continuous Viewpoint

Weinan E, Chao Ma, Lei Wu

arXiv:1912.12777v2math.NAcs.NAmath.OCstat.ML

TL;DR

Machine learning lacks fundamental principles for designing models and understanding their performance. The paper formulates machine learning continuously, recovering familiar models as discretizations and deriving new models and algorithms; it also frames neural networks as simple particle-method discretizations.

  • Problem

    Machine learning lacks fundamental guiding principles for designing models and understanding performance, while practical techniques often require heavy parameter tuning and can be fragile.

    Source: §1 · p. 3

  • Method

    The paper formulates machine learning through function representations, loss functionals, and training dynamics, treating it as a calculus-of-variations or PDE-like problem discretized by numerical algorithms.

    Source: §1 · p. 2

  • Results

    The framework recovers two-layer neural networks and ResNets in scaled form through particle-method discretizations and yields new models and algorithms, including flow-based random features, smoothed particle methods, and spectral methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Neural networks are the simplest particle-method discretizations of the simplest continuous gradient-flow models for integral-transform and flow-based representations.

    Source: §8 · p. 37

    Approximation errors in the function and sampling errors can interact in population-risk computations, and this interaction has not been systematically investigated.

    Source: §3.3 · p. 9

Abstract

from arXiv · show

We present a continuous formulation of machine learning, as a problem in the calculus of variations and differential-integral equations, in the spirit of classical numerical analysis. We demonstrate that conventional machine learning models and algorithms, such as the random feature model, the two-layer neural network model and the residual neural network model, can all be recovered (in a scaled form) as particular discretizations of different continuous formulations. We also present examples of new models, such as the flow-based random feature model, and new algorithms, such as the smoothed particle method and spectral method, that arise naturally from this continuous formulation. We discuss how the issues of generalization error and implicit regularization can be studied under this framework.

1 Introduction

The paper proposes a continuous framework for machine learning to provide guiding principles for model and algorithm design. It treats learning as a variational or PDE-like problem whose discretizations recover familiar models and motivate new ones.

1 Introduction

  • The framework has three components: a function representation, a loss functional, and training dynamics.It focuses on expectation-based representations and uses model A gradient flow in L2 for non-conserved parameters and model B gradient flow in the Wasserstein metric for conserved parameters.Source: §1 · pp. 2, 4
  • Different numerical algorithms discretize the continuous variational or PDE-like models, recovering two-layer neural networks and ResNets in scaled forms.The paper also proposes a flow-based random feature model, a new class of transform-based models, smoothed particle methods, and spectral methods.Source: §1 · p. 2
  • The authors conjecture that over-parameterized models may behave better because they provide more accurate discretizations of continuous gradient flow.They also suggest that very deep fully connected networks without residual connections lack continuum limits and can suffer exploding gradients.Source: §1 · pp. 2–3
  • The continuous framework builds on prior work but is presented as the first systematic exploration of the continuous viewpoint.The paper describes its approach as philosophically similar to formulating a continuous variational problem before discretization in PDE-based image processing.Source: §1 · p. 3
  • Machine learning lacks fundamental guiding principles for designing models and understanding performance, leaving many methods ad hoc and sensitive to hyperparameter choices.The paper seeks a reasonably simple and transparent framework, while recognizing that machine learning must address very high dimensions.Source: §1 · pp. 3–4

2 Representations of functions

The paper favors function representations expressible as expectations because they support dimension-independent approximation rates. It develops integral-transform and flow-based representations, including compositional forms and a general flow representation needed for residual-network continuum limits.

2 Representations of functions

  • Expectation-based representations are favored for high-dimensional settings because independent sampling can yield dimension-independent approximation error, unlike grid-based approximation with a curse-of-dimensionality rate.The passage contrasts an exponent containing 1/d for grid-based approximation with the dimension-independent rate for sampling from π.Source: §2 · pp. 4–5

2.1 Integral-transform based representation

  • Generalized ridgelet transforms use nonlinear scalar functions and can be extended to higher co-dimension structures and compositional representations with multiple hidden layers.The paper notes that these representations are better suited to high-dimensional situations.Source: §2.1 · p. 6

2.2 Flow-based representation

  • Flow-based representations generate trial functions through a continuous dynamical system, and the more general forms are needed to capture residual neural networks' continuum limit.The flow map at time 1 maps x to z(1); the construction can allow a change of dimension.Source: §2.2 · p. 7

3 The optimization problem

The section formulates several learning and scientific tasks as continuous optimization problems, then describes their risks, objectives, and constraints. It also notes that sampling-based estimation in variational problems can interact with approximation error.

3 The optimization problem

  • The section frames supervised learning, dimension reduction, variational problems, and nonlinear parabolic PDEs as optimization tasks with distinct objectives.It introduces these as examples of machine-learning tasks formulated through continuous calculus-of-variations problems.Source: §3 · p. 7

3.1 Supervised learning

  • Supervised learning minimizes population risk against a target function, while finite samples require working with empirical risk.The loss can use a convex function, and labels may be noisy.Source: §3.1 · p. 8

3.2 Dimension reduction

  • Dimension reduction assumes data concentrate on a lower-dimensional set and seeks encoder and decoder functions that minimize reconstruction error.The target dimension is assumed known.Source: §3.2 · p. 8

3.3 Calculus of variations

  • For variational problems such as finding a quantum ground state, sampling from the induced distribution estimates the objective, but approximation and sampling errors may interact.The interaction has not been systematically investigated.Source: §3.3 · pp. 8–9

3.4 Nonlinear parabolic PDEs

  • High-dimensional nonlinear parabolic PDEs can be posed variationally, with backward stochastic differential equation constraints forming the starting point of the Deep BSDE method.The section identifies analysis of these variational problems as a major task in mathematical machine learning.Source: §3.4 · p. 9

4 Gradient flows

This section develops gradient-flow formulations for population and empirical risk, distinguishing conserved probability distributions from non-conserved coefficients and extending the analysis to flow-based models, with connections to control theory and conditions and guarantees for particular models. It also concludes a proof by taking expectation over x and notes that rigor requires existence and uniqueness of the limit of GMM (72), whose details are omitted.

4.1 Conservative and non-conservative gradient flows

  • For conserved distributions and non-conserved coefficients, the framework recovers two-layer-network gradient descent in the mean-field limit [61] and the continuous gradient flow for random feature models.The probability distributions π or ρ are conserved, while the coefficient a is not.Source: §4.1 · pp. 10–11

4.2 Pontryagin’s maximum principle for flow-based models

  • Pontryagin’s maximum principle gives a necessary condition for an optimal flow-based model: states evolve forward in time, while co-states evolve backward.The model parameters act as controls, and the state, co-state, and Hamiltonian are the corresponding control-theory quantities.Source: §4.2 · p. 12

4.3 Flow-based random feature model

  • In the flow-based random feature model, π is fixed and V has rank d + 1; under the stated approximation assumptions, any stationary point with finite expected coefficient norm is a global minimum.The assumptions include continuity of the target function and a universal approximation property for the feature family.Source: §4.3 · pp. 12, 14
  • The flow-based random feature model’s gradient equations couple different τ values through the state dynamics and include a dissipation result for the gradient flow.The supplied passages identify dissipation and stationary-point results but do not provide their formulas.Source: §4.3 · pp. 14–15

4.4 Gradient flow for the flow-based neural networks

  • For flow-based neural networks, the gradient flow over probability-measure parameters is obtained as the generalized minimizing-movements limit, yielding a 2-Wasserstein gradient flow for minimizing the Hamiltonian.The parameter space consists of one-parameter families of probability measures.Source: §4.4 · pp. 17, 19
  • The proof concludes by taking expectation with respect to x.Source: §4.4 · p. 19
  • Establishing existence and uniqueness of the limit of GMM (72) is needed to make the argument rigorous, but the details are left to the interested reader.The paper characterizes the omitted proof as lengthy but straightforward.Source: §4.4 · p. 19

5 Discretizations

The section treats machine learning models as continuous variational or flow formulations that can be discretized, focusing on parameter-space discretization of training dynamics. It recovers scaled two-layer networks and develops smoothed-particle and integral-transform-based algorithms.

5 Discretizations

  • The section focuses on discretizing parameter space and the training dynamics, rather than discretizing the loss variational problem and then applying an optimization algorithm.Discretization in real space can be more difficult for reinforcement learning and PDE-solving problems, which the section leaves for future work.Source: §5 · p. 19

5.1 Recovering the two-layer neural network model

  • Parameter-space discretization of the training dynamics recovers the continuous-time gradient descent dynamics of a scaled two-layer network.The network output includes a factor of 1/m, and the continuous flow equation also holds for a finite set of neurons.Source: §5.1 · pp. 20–21

5.2 A smoothed particle method

  • For the ReLU integral-transform model, smoothing the particle method yields a two-layer network with a smoothed ReLU activation whose uniform difference from ReLU is O(h).The paper also formulates a smoothed particle discretization for the flow-based model; evaluating its terms for a practical algorithm is deferred.Source: §5.2 · pp. 21–22

5.3 A new algorithm for integral transform-based models

  • The new integral-transform algorithm uses a modified gradient flow because designing a particle method for the combined model A and model B dynamics is difficult.Its particle dynamics are evaluated in preliminary numerical experiments.Source: §5.3 · pp. 23–24
  • Testing errors for both an RKHS target and a target outside the RKHS decrease at rate O(1/t) during training.The experiments use 100 online samples per step, so there is no generalization gap in this setup.Source: §5.3 · p. 24, Figure 2 · p. 24

6 The generalization error

The section studies generalization error by obtaining a priori estimates for discretized and continuous training dynamics. It illustrates both routes with random-feature models, while noting that continuous-model discretization error requires further numerical-analysis work.

6 The generalization error

  • The section illustrates two routes to generalization estimates—bounds for discretized gradient descent and bounds for the continuous gradient flow—using random-feature models.The discretized route recovers results from [14] with simpler arguments.Source: §6 · p. 25

6.1 Analyzing the discretized model

  • The discretized-model analysis combines training-error and approximation-error estimates to obtain a priori bounds for gradient-descent solutions.The stated result assumes m ≥ log^2(n/δ) and holds with probability 1−δ.Source: §6.1 · pp. 26–28

6.2 Analyzing the continuous model

  • The continuous-model approach is explored because standard theory relies on convexity, which is difficult to extend to more complicated settings; PDE techniques offer an alternative analysis route.The passage cites a global-convergence result for two-layer-network gradient flow proved by analyzing the PDE.Source: §6.2 · p. 29
  • The generalization-error analysis separates discretization error, generalization gap, and continuous-problem training error; the latter two require a priori estimates for the gradient flow.The treatment of discretization error is postponed because it requires substantial numerical-analysis machinery.Source: §6.2 · pp. 29–30

7 An example

The example analyzes a one-dimensional conservative gradient flow for an integral-transform model, then examines convergence and frequency-dependent behavior analytically and numerically. Its results distinguish global convergence for a uniform target from local convergence for general targets and show that frequency ordering can fail.

7 An example

  • The example studies a one-dimensional conservative gradient flow for an integral-transform model, with a fixed target distribution and a convolutional kernel assumed positive definite and sufficiently smooth.The kernel is three-times differentiable with bounded derivatives.Source: §7 · p. 30

7.1 Global convergence for uniform target distribution

  • For a uniform target distribution and differentiable initial density, the gradient flow converges globally to the target in W2.The proof uses relative entropy as a Lyapunov function.Source: §7.1 · pp. 30–31

7.2 Local convergence for the general case

  • For general targets, initialization sufficiently close to the target yields local convergence to the global minimum at an O(1/t) rate under the theorem’s stated conditions.The proof assumes a decay condition on the kernel’s Fourier coefficients; the authors say it can be relaxed with faster decay imposed on the target and initial distributions.Source: §7.2 · pp. 32–33

7.3 Numerical results

  • With 100 uniformly sampled data points, the empirical loss appears to converge to zero while the density’s L2 norm stays bounded.Figure 4 plots empirical risk and the solution’s L2 norm along the gradient-descent path.Source: §7.3 · p. 35, Figure 4 · p. 35

7.4 The frequency principle

  • The frequency principle holds in the illustrated case, but need not hold generally: the target’s spectral energy and the kernel spectrum can make a high-frequency component converge faster.For the illustrated exception, frequency k = 5 converges faster than k = 1; Figure 6 shows the case where the principle does not hold.Source: §7.4 · pp. 36–37

8 Discussions

The discussion frames machine learning through continuous representations, variational problems, and gradient flows, with familiar neural networks arising as particular discretizations. It proposes design principles for robust models and algorithms while noting that very deep fully connected networks lack nice continuum limits.

8 Discussions

  • The continuous viewpoint focuses on function representations, variational problems, and continuous gradient flows; features and neurons arise in special discretizations.It treats shallow networks and ResNets as the simplest particle-method discretizations of gradient flows for integral-transform and flow-based representations, respectively.Source: §8 · p. 37
  • The proposed design principles call for expectation-based target representations, well-behaved risk functionals and gradient flows, and stable long-time discretizations.The risk functionals need not be convex, but should share many features of convex functionals.Source: §8 · p. 38
  • The authors suspect that following these principles will yield robust models and algorithms, unlike current models that can depend sensitively on hyperparameters.This robustness is presented as a suspicion, not an established result.Source: §8 · p. 38
  • The passage notes that some subtleties in current machine-learning algorithms can be appreciated from a continuous viewpoint, and says very deep fully connected networks should cause problems because they lack nice continuum limits.Source: §8 · p. 38

Related papers

Loading 1912.12777v2…