PaperlayerView exact revision on arXiv

Source-linked AI summary

The Hamilton-Jacobi Theory of Deep Learning

Jose Marie Antonio Miñoza, Erika Fille T. Legara, Christopher P. Monterola

arXiv:2605.28983v2cs.LGcs.AImath.DSmath.RTphysics.comp-ph

TL;DR

The paper addresses how standard trained neural networks relate to underlying equations. It identifies LSE layers exactly with Hopf–Cole propagators of viscous Hamilton–Jacobi equations and unifies network, tropical, PDE, and convex views through ε. The resulting framework gives quantitative claims about generalization, robustness, residual-network backpropagation, scaling, and attribution.

  • Problem

    The paper asks what equation a trained neural network solves, reversing the conventional direction of designing networks to approximate prescribed PDE solutions.

  • Method

    The paper uses an exact LSE–Hopf–Cole identity and a single deformation parameter ε to connect neural layers with viscous Hamilton–Jacobi equations, tropical algebra, and convex optimization.

  • Results

    The framework yields a minimax generalization rate O(n^-1/(d+2)), ε-controlled adversarial robustness, residual-network backpropagation as a co-state equation, and an O(N) attribution influence function with fold bifurcations.

  • Takeaways & Limitations

    A trained network can be read as a Hamilton–Jacobi equation whose weights encode initial data and whose input is the point where the solution is evaluated.

  • Takeaways & Limitations

    The exact Hopf–Cole identity is limited to the quadratic Hamiltonian class; outside that class, a nonzero reaction term remains and the identity fails.

Abstract

from arXiv · show

In this paper, training a neural network is identified, exactly, as a search through Hamilton--Jacobi initial-value problems: each gradient step selects the initial data of a viscous Hamilton--Jacobi equation whose Hopf--Cole propagator best fits the observations; at inference, the input is the spatial point at which that solution is evaluated and the initial condition is already encoded in the weights. The correspondence is exact for log-sum-exp layers, with ReLU, sigmoid, SiLU, and GELU each an exact limit, gradient, or moment of the same object, and exact in composition across depth and width, with a quantified error at finite depth that vanishes in the joint limit. It is structural for residual networks, transformers, and recurrent networks (RNNs, LSTMs, SSMs), each discretizing the same class of equations, at a named and quantified approximation error. A single deformation parameter $\varepsilon$ unifies all four perspectives (network, tropical algebra, viscous PDE, convex optimization) in a commutative diagram closed under Lipschitz conditions. Quantitative consequences include: the minimax optimal generalization rate $O(n^{-1/(d+2)})$ for fixed $t$; adversarial robustness controlled by $\varepsilon$; backpropagation as the co-state equation of the Hamiltonian system for residual networks (Pontryagin Maximum Principle); scaling exponents consistent with data intrinsic dimension via PDE quadrature; and a closed-form $O(N)$ influence function (softmax attribution weights $π_j$) whose entropy landscape undergoes fold bifurcations as $\varepsilon$ increases, each merging attribution basins.

1 Introduction

The paper identifies trained neural networks with Hamilton–Jacobi equations through a single deformation parameter ε, unifying tropical algebra, viscous PDEs, convex optimization, and standard architectures. This correspondence yields quantitative consequences for generalization, robustness, training dynamics, and attribution.

  • Core correspondence: A trained network is interpreted as a Hamilton–Jacobi equation whose initial data are encoded by its weights and whose solution is evaluated at the input.The framework reverses the usual PDE-to-network direction: training searches over initial conditions for the equation that best fits observations.
  • Core correspondence: ε simultaneously parametrizes the softmax temperature, PDE viscosity, and convex regularization, connecting tropical and smooth computation.At ε = 0 the object becomes tropical; for ε > 0 it is represented by ordinary arithmetic and a viscous PDE solution operator.
  • Architectural scope: LSE layers correspond exactly to Hopf–Cole solutions, compose structurally across depth, and extend to transformer attention and recurrent architectures.The tropical limit recovers Hopf–Lax inf-convolution, while ResNets and recurrent networks discretize ODE characteristics with architecture-dependent viscosity.
  • Quantitative consequences: O(n^-1/(d+2)) is the minimax generalization rate for fixed t, while adversarial robustness is certifiably controlled by ε and backpropagation is a residual-network co-state equation.The framework also provides convexity and Lipschitz-inflation guarantees and extends to learnable quadratic Hamiltonians.
  • Quantitative consequences: The softmax weights π_j form a closed-form O(N) influence function with exact label sensitivity, and attribution entropy undergoes fold bifurcations as ε increases.Each bifurcation merges attribution basins, with saddle points marking attribution transitions.

2 Background

The background presents log-sum-exp as a smooth, entropy-linked relaxation of max and places it within tropical algebra, Hamilton–Jacobi theory, and ultradiscretization. These views are connected through the temperature or viscosity parameter ε.

  • Log-sum-exp and convex duality: LSE_ε is convex and smooth for ε > 0, has softmax gradient, and has negative entropy as its Legendre–Fenchel conjugate.It is simultaneously a Gibbs log-partition function, a smooth max relaxation, and a heat-equation solution operator via Hopf–Cole.
  • Tropical algebra: The tropical semiring replaces addition with max and multiplication with addition, with tropical matrix multiplication acting by max-plus aggregation.The structure uses −∞ and 0 as the respective identities.
  • Hamilton–Jacobi PDEs: Viscous Hamilton–Jacobi equations use ε as viscosity; Hopf–Cole linearizes the quadratic Hamiltonian case to the heat equation.For general convex Hamiltonians, the substitution produces a nonlinear equation, while the inviscid case is characterized by Hopf–Lax solutions.
  • Ultradiscretization: Ultradiscretization sends smooth addition to max-plus addition as ε approaches zero, while increasing ε reverses the lift from tropical to smooth systems.The same principle is applied here to neural networks.

3 From Neural Networks to Max-Plus Algebra

The LSE activation is a smooth deformation of tropical max, with ε controlling the transition between a soft, distributed computation and a hard max-affine computation. The limit yields exact tropical algebra and ReLU-like operators.

  • Deformation to tropical algebra: ε → 0 gives the semiring homomorphism from ordinary arithmetic to tropical arithmetic, where addition becomes max and multiplication becomes +.For finite ε, LSE remains bounded between max_i x_i and max_i x_i + ε log m.
  • Two network regimes: At ε > 0, every neuron contributes through softmax weights and the layer is an entropy-regularized optimization.At ε = 0, a single neuron dominates and the layer becomes a max-affine spline operator.
  • Two network regimes: The ε = 0 layer partitions inputs into polyhedral regions, functioning as a max-affine spline operator analogous to a decision tree.Softmax weights at positive ε continuously relax the one-hot region indicator.
  • Two network regimes: ReLU arises at ε = 0 through a two-neuron comparison max(W_j · x, 0), placing it within the tropical semiring operation.The result follows from the hard-max limit of the LSE construction.

4 Neural Networks as Hamilton–Jacobi Equations

The paper establishes an exact algebraic identity between LSE network layers and Hopf–Cole solutions of viscous Hamilton–Jacobi equations. It then interprets weights, biases, temperature, attention, and layer composition within this PDE framework.

  • Exact layer identity: An LSE network layer encodes the Hopf–Cole solution exactly under a discrete measure, rather than approximating the PDE solution.The identity holds at fixed layer depth and identifies the network output with the viscous Hamilton–Jacobi solution up to an x-independent normalization constant.
  • Exact layer identity: The layer dictionary maps weights to initial-data support points, biases to transport costs, ε to viscosity, and width N to measure discretization.The relation W_j = y_j/(2t) makes each neuron weight a coordinate of an initial-data support point.
  • Training interpretation: Training searches over initial conditions for the viscous Hamilton–Jacobi equation, so no external PDE residual loss is required.The PDE is built into the architecture, and the spatial residual is identically zero at fixed layer depth.
  • Attention: Transformer attention is an expected value under the same Gibbs measure, while its zero-temperature limit becomes hard argmax attention.L2 attention with negative squared-distance logits gives an exact Hopf–Cole evaluation with no approximation.
  • Depth and extensions: Composing layers corresponds structurally to composing Hamilton–Jacobi semigroups, with each layer discretizing a time step under its own discrete measure.The network’s depth and widths determine the sequence of semigroup evaluations and their finite-depth approximation behavior.
  • Depth and extensions: The exact Hopf–Cole identity extends from isotropic quadratic H to learnable positive-definite quadratic Hamiltonians through an anisotropic quadratic shift.For Hamiltonians outside the quadratic class, the substitution leaves a nonzero reaction term and the identity fails.

5 The Tropical Limit and Convex Optimization

As ε approaches zero, smooth log-sum-exp layers collapse to tropical max-plus operators and Hopf–Lax solutions, linking neural computation to convex optimization. The same Hamilton–Jacobi characteristic structure extends to residual, recurrent, and state-space architectures.

  • Tropical limit: At ε = 0, the Hopf–Cole predictor converges to the Hopf–Lax formula, represented simultaneously as a tropical inner product, MASO, linear program, and piecewise-affine function.The breakpoints occur at the support points {y_j}.
  • Architectural correspondences: ResNet recurrences converge to Hamilton–Jacobi characteristic ODEs as step size h approaches zero and depth grows with Lh fixed.The corresponding Hamiltonian is H(x,p) = p · F(x,θ).
  • Architectural correspondences: RNNs, SSMs, and LSTMs fit the same characteristic framework, with recurrent dynamics, linear state-space evolution, and gated selectors supplying architecture-specific Hamiltonians.LSTM gates are gradients of two-neuron LSE layers, while SSMs have a max-plus tropical limit.
  • Architectural correspondences: Viscosity is global in feedforward networks, input-dependent in SSMs, and gated per cell in LSTMs.The LSE class is closed under composition, affine input transformations, and residual connections.

6 The ε Parameter: A Unifying Deformation

A single deformation parameter ε connects neural-network layers, tropical algebra, viscous Hamilton–Jacobi equations, and convex optimization. The resulting dictionary also supports infinite-width and overparameterization analyses.

  • Four perspectives: ε indexes four descriptions of the same object: LSE or max network computation, tropical or ordinary algebra, viscous or inviscid HJ equations, and entropic regularization or linear programming.The ε = 0 endpoint gives the tropical, inviscid, vertex-based description.
  • Four perspectives: At ε > 0, the layer is smooth and entropy-regularized; at ε = 0, it becomes a max-plus or linear-program vertex computation.The same parameter therefore controls the deformation from soft to hard computation.

7 The Commutative Diagram

The commutative diagram identifies finite-width network layers with Hamilton–Jacobi semigroup operations and their tropical limits. Under Lipschitz conditions, width and viscosity limits commute, with quadrature error quantified explicitly.

  • Diagram structure: Composing L layers corresponds to composing Hamilton–Jacobi semigroup operators, while finite networks pass point evaluations between layers rather than integrating the full function.Finite-depth and joint-limit exactness are quantified in the cited theorems.
  • Quantified limits: As N approaches infinity, the discrete solution converges to the continuous solution at rate O(N^-1/d).This is the quadrature approximation rate under the stated Lipschitz framework.
  • Quantified limits: As ε approaches zero, the viscous solution converges pointwise to the inviscid solution.The two routes—first increasing width or first taking the tropical limit—yield the same final limit.
  • Diagram structure: The diagram commutes by combining Maslov dequantization with the identification of LSE layers as discrete-measure PDE solutions.The exact network correspondence applies to the quadratic Hamiltonian and its anisotropic extension.
  • Assumptions: The commutation claims require Lipschitz conditions because softmax concentration affects quadrature convergence.The semigroup facts extend abstractly to Lipschitz g and convex superlinear H, whereas the exact LSE identity is tied to quadratic H.

8 Consequences of the Correspondence

The correspondence yields quantitative statements about approximation, generalization, robustness, training dynamics, extrapolation, intrinsic-dimension scaling, and attribution geometry. These consequences include both exact identities and bounds whose scope depends on width, viscosity, depth, and regularity assumptions.

  • Generalization: O(n^-1/(d+2)) is the fixed-t generalization rate obtained by balancing PDE quadrature and statistical error.The corresponding optimal width is N* ≍ (n/M^2)^(d/(d+2)).
  • Generalization: ε* ≍ N^-1/d balances quadrature under-resolution against viscosity over-smoothing, prescribing N ≍ δ^-d for target error δ.For depth-L networks, semigroup composition adds a factor of L to the composition error.
  • Robustness: Increasing ε suppresses curvature and widens this particular certified robustness bound, while viscosity smooths the solution against pointwise perturbations.The curvature dependence also describes the robustness–expressiveness tradeoff.
  • Robustness: The robustness certificate also has an ε-free radius r = τ/∥W∥2,∞ that dominates the curvature-based certificate for every ε.Equality holds only as ε approaches infinity.
  • Training dynamics: Backpropagation is the reverse-time co-state equation of the Hamiltonian system for residual networks.The forward and backward equations together form the complete Hamiltonian flow, and weight updates satisfy the Pontryagin Maximum Principle condition.
  • OOD behavior: Outside the diffusion radius √(2εt), outputs are exponentially close to the dominant neuron’s linear extrapolation and do not depend on the training distribution at that point.The paper characterizes this structural out-of-distribution extrapolation as hallucination.
  • Scaling: L(N) ≲ N^-1/d_eff under the manifold hypothesis, making the population-loss exponent inversely proportional to intrinsic data dimension.The result is an upper bound, and the paper distinguishes its order-0 Lipschitz rate from stronger smooth-target rates.
  • Attribution: As ε increases, attribution-entropy fold bifurcations merge neighboring attribution basins by annihilating a saddle and an adjacent minimum.Under the generic assumptions, the small-ε landscape has N local minima at the support points.

9 Numerical Experiments

Numerical experiments verify the core Hopf–Cole and attention identities to machine-level precision and confirm the predicted quadrature convergence rate in a well-resolved regime.

  • ∼10^-16 identity error confirms LSEε(Wx+b) matches the Hopf–Cole expression to machine precision.The result is reported in Table 2.
  • 0 error confirms the attention identity across dimensions d ∈ {4, . . . , 64}.This is reported in Table 3.
  • O(N^-1/d) ℓ∞ error decay confirms the quadrature rate for Lipschitz g(y) = |y| at fixed (ε, t).The confirmation applies in the well-resolved regime.

10 Conclusion

The conclusion presents neural networks, tropical algebra, viscous Hamilton–Jacobi PDEs, and mechanics as manifestations of a shared ε-deformation framework. It connects this structure to classical simulation, architecture design, and limiting regimes across learning and physics.

  • ε → 0 concentrates computation on the minimum-cost neuron, recovering the tropical or classical limit through Laplace’s method.This parallels the emergence of classical trajectories from path integrals.
  • A single ε identifies LSE networks, Gibbs partition functions, and imaginary-time Schrödinger propagators as the same mathematical object.The parameter simultaneously acts as softmax temperature, PDE viscosity, quadrature regularization, and quantization scale.
  • ResNet recurrence implements repeated local Hamilton–Jacobi characteristic steps, with backpropagation linked to the associated co-state dynamics.The architecture is interpreted as a cellular automaton discretizing characteristic evolution.
  • The LSE framework provides a positive-measure classical simulator for Hamilton–Jacobi initial-value problems, with an O(N^-1/d) approximation rate for Lipschitz data.The rate is described as optimal within the stated setting.
  • The framework identifies ReLU and related piecewise-linear theories with the ε = 0 tropical limit and connects HJ smoothing to optimization landscapes through a role reversal.In the present framework, the input is the spatial variable and weights encode initial data.

C Machine Learning Tasks as Initial-Value Problems

This section casts standard learning tasks as Hamilton–Jacobi initial-value problems whose architecture determines the PDE class and whose training selects initial conditions. It extends the correspondence across regression, classification, kernels, sequences, encoder–decoders, and mean-field optimization.

  • Architecture as PDE, Output as Solution: A network architecture can encode the PDE directly, making its output a solution value without requiring a residual loss.For LSE layers, the PDE residual is zero by construction before training.
  • Architecture as PDE, Output as Solution: Training searches over discrete initial data, selecting the viscous HJ equation whose propagated solution best fits observations.The weights encode the initial data while the architecture determines the Hamiltonian.
  • Regression: Regression evaluates the Hopf–Cole solution at the input and trains initial data to approximate the target function.The loss is minimized over inputs while selecting the initial data g.
  • Classification: Classification uses softmax probabilities as Gibbs weights over class scores, with the ε → 0 limit producing one-hot predictions and sharper tropical boundaries.The predicted class is the tropical argmax.
  • Kernel Machines: Kernel estimators are Hopf–Cole pre-images, with ε = σ^2/2 and the ε → 0 limit recovering nearest-neighbor classification.The transformed kernel estimator corresponds to the linearized variable v.
  • Time Series and Sequences: RNNs, LSTMs, and SSMs interpret sequence index as PDE time and learn Hamiltonian characteristics matching observed trajectories.Transformers analogously use token embeddings as spatial variables and depth as PDE time.
  • Encoder–Decoder: Encoder–decoder composition follows the PDE semigroup, with the bottleneck representing the intermediate PDE state.The composed propagators yield total evolution time t1 + t2.
  • Training Dynamics: Mean-field SGD converges to a Wasserstein gradient flow over initial-data measures, while finite-N particle approximation incurs O(N^-1/d) error.The co-state equation supplies the optimal update direction in the corresponding Hamiltonian system.

E Proofs

The paper derives exact and approximate correspondences between neural-network layers, Hamilton–Jacobi propagation, and semigroup composition, with errors controlled by width, depth, and viscosity. These correspondences yield generalization, scaling, robustness, and optimization consequences.

  • No approximation is made in the gated-activation identity; the correspondence is algebraic.
  • The quadratic parameterization Wj = yj/(2t) uniquely exposes the transport cost |x−yj|2/(4t) inside each neuron.
  • At ε = 0, the Hopf–Lax formula becomes nearest-neighbor retrieval, while finite ε assigns soft weights to all memories.
  • Multilayer Composition: Composing layers corresponds to applying heat-equation semigroups for additive total time T = t1 + ··· + tL.
  • Multilayer Composition: Finite-depth composition has error linear in depth and inherits the single-layer rate O(N −1/d).
  • Multilayer Composition: The depth-L network converges to the continuous HJ semigroup as L →∞ with tℓ = T/L and sufficient width Nℓ≳t−dℓ.
  • Generalization: For fixed t, balancing width and sampling terms recovers the minimax rate O(n−1/(d+2)).
  • Optimization: Backpropagation is the discrete co-state equation paired with the forward residual-network state equation under the Pontryagin Hamiltonian system.

G Numerical Details

The numerical study combines analytical identity checks, trained-network experiments, and scaling and robustness tests across synthetic and image data. Results verify the HJ identities to floating-point precision, observe predicted width scaling, and confirm Hessian-bound behavior under the stated PCA settings.

  • Experimental setup: All trained networks use Adam with fixed initialization conventions and viscosity ε = N^-1/d, except Figure 4’s fixed ε = t = 1 quadrature check.The experiments vary optimizer steps, learning rates, widths, seeds, dimensions, and datasets by task.
  • Identity verification: Analytical identity checks verify transformer attention and LSE-FFN correspondences to floating-point precision without optimizer training.Experiment D checks attention analytically, while Experiment G uses 500 trials per dimension and reports machine-precision agreement.
  • Scaling tests: O(N^-1/d) ℓ∞ error decay appears for Lipschitz g(y) = |y| at fixed (ε, t) = (1, 1), confirming the well-resolved quadrature regime.The figure’s headline generalization-gauge regime t ≍ ε ≍ N^-1/d is treated analytically rather than by this fixed-parameter plot.
  • Scaling tests: Fitted exponents α̂ ≈ 1/d for Adam-trained LSE networks confirm the predicted intrinsic-dimension scaling across d ∈ {1, 2, 4}.The target is g(y) = ∥y∥ and the plotted quantity is test RMSE versus width N.
  • Robustness checks: The Hessian bound is never violated for the reported SGD-trained synthetic network and for MNIST and CIFAR-10 across the tested ε values.Image verification uses N = 128 neurons with PCA dimensions d = 50 for MNIST and d = 64 for CIFAR-10, over 300 random test images.

H Scope of the Correspondence: Universality, Closure, and Extensions

The correspondence supports universality and closure for deep LSE HJ solutions, while extending structurally beyond the exact quadratic Hopf–Cole case. Non-convex Hamiltonians remain bounded by convexification error, and depth enlarges the function class without removing that intrinsic gap.

  • Universality: Deep LSE HJ solutions approximate every continuous function on compact sets, making the HJ class as general as continuous functions when depth is at least two.Proposition H.1 supplies arbitrary uniform accuracy, while the depth condition resolves the convexity restriction of single layers.
  • Closure: The HJ function class is closed under layer composition, affine input transformations, and residual connections, with composition retaining the class up to O(h) discretization error.Each layer is exact, while characteristic-based composition introduces the stated finite-step truncation error.
  • Extensions: For convex Hamiltonians, the framework extends to scalar conservation laws, eikonal equations, and linear transport by encoding the Hamiltonian in weights and viscosity in ε.The extension relies on Hopf–Lax or inf-convolution representations.
  • Non-convex Hamiltonians: For general non-convex H, Hopf–Cole linearization is exact only for quadratic Hamiltonians; otherwise the substitution leaves a nonlinear reaction term.The exact class includes H(p) = |p|² and anisotropic quadratic forms p⊤Aθp.
  • Non-convex Hamiltonians: The non-convex construction solves the convexified problem H** with error bounded by t · sup|H − H**|, vanishing when H is convex.LSE-type kernels are exact for H**, while small non-quadratic perturbations produce identity failure linear in non-convexity and evaluation time.
  • Non-convex Hamiltonians: Depth enables non-convex target representations, but cumulative convexification error remains O(t · ∥H − H**∥∞) and is independent of depth.Each layer is convex in its own input, while compositions with L ≥ 2 need not be globally convex.
  • Kernel and architectural interpretations: The N → ∞ limit connects LSE networks to continuum heat-kernel integrals, while parameter sharing in CNNs corresponds to shift-invariant initial-data measures.The same viscous HJ equation is evaluated position-wise across receptive fields.
  • Activation extensions: GELU is placed in the same propagator family through Gaussian convolution, and its ε → 0 limit recovers ReLU.The correspondence status table distinguishes solution, gradient, and moment/measure classes for common activations.

I Intrinsic Dimension from Published Scaling Curves

Published scaling curves are used to estimate effective intrinsic dimension through deff = 2/α, with substantial variation across domains. These estimates are directional and can reflect training and data-coverage effects, not geometry alone.

  • Dimension estimates: deff = 2/α converts published scaling exponents into estimated intrinsic dimensions for squared-error or cross-entropy losses.The conversion follows LCE(N) ≲ N^-2/deff.
  • Cross-domain variation: deff ≈5.7 is recovered for compute-optimal language scaling, compared with deff ≈26.3 in the under-trained Kaplan regime.The difference is attributed to insufficient training in the Kaplan setting.
  • Interpretation: Empirical α values are order-of-magnitude indicators because they conflate approximation, optimization, and data-coverage effects.The Kaplan exponent specifically reflects under-training rather than the data manifold alone.
  • Cross-domain variation: deff varies across domains in a pattern consistent with the manifold hypothesis: mathematics is lower-dimensional than language, while video is intermediate.The passage characterizes these as structured differences in degrees of freedom.

J Actionable Design Principles

The framework turns temperature, architecture, scaling, and training dynamics into theorem-grounded design choices. It links ε to approximation and robustness, interprets architectures as discretizations, and identifies backpropagation with adjoint dynamics.

  • Temperature: ε* ≍ N^-1/d is the minimax-optimal viscosity balancing approximation and estimation error.The resulting excess-risk form is O(N^-1/d + M N/n).
  • Temperature: Increasing ε monotonically decreases perturbation sensitivity, providing a certified robustness control without retraining.Lower ε toward ε* sharpens discrimination, while higher ε smooths responses.
  • Architecture: Architecture choice is framed as a discretization strategy for the same Hamilton–Jacobi equation rather than solely an empirical design decision.Feedforward, residual, attention, and recurrent architectures correspond to different discretizations.
  • Architecture: Tasks with temporal ordering favor process discretizations, static-distribution invariants favor measure discretization, and query-dependent weighting favors attention.These prescriptions distinguish recurrent or residual, feedforward, and attention mechanisms by task structure.
  • Training dynamics: Backpropagation is the co-state equation of the Hamiltonian system associated with the Hamilton–Jacobi PDE.The paper identifies reverse-mode autodifferentiation with the adjoint method for PDE-constrained optimization.
  • Scaling: deff = 2/α and N ∝ n^d/(d+2) provide scaling estimates and compute-allocation guidance before large training runs.The prescription derives from the framework’s approximation-rate analysis.

K Implications for Large-Scale Training, Alignment, and Continual Learning

The framework argues that large-scale training is increasingly constrained by model-side approximation and intrinsic dimension, while alignment and continual learning can be expressed through initial-data selection and measure coverage. Its scope remains limited by training-selection guarantees, dimensionality, and architecture-specific exactness.

  • Large-scale training: At fixed dataset size, adding data cannot reduce the estimation floor, shifting the bottleneck toward width and intrinsic dimension.The gain from doubling n becomes negligible once n ≫ N^(deff+2)/deff.
  • Large-scale training: Halving approximation error requires 2^deff more neurons, making direct reduction of deff more efficient than brute-force width scaling.At deff = 26.3, each halving costs roughly 8.3 × 10^7 more model parameters.
  • Large-scale training: Architectural symmetries and data curation are proposed as levers for reducing effective dimension.Translation equivariance can collapse a d-dimensional problem to receptive-field dimension K, while curated domains exhibit lower deff values.
  • Alignment: Alignment is formulated as designing a risk whose minimizing initial-value problem encodes human-valued initial data.The framework also characterizes hallucination structurally in sufficiently out-of-distribution regions.
  • Continual learning: Reassigning k of N neurons covering an earlier task worsens its approximation rate from O(N^-1/deff) to O((N-k)^-1/deff).This quantifies catastrophic forgetting as reduced coverage of the earlier task’s support measure.
  • Limitations: The exact Hopf–Cole identity is restricted to quadratic Hamiltonians, while non-quadratic cases receive approximation or structural correspondences.The quadratic class is stated to be maximal for exact identities.
  • Limitations: The minimax rate O(N^-1/d) retains the curse of dimensionality, and deff inferred from scaling exponents is not a certified support-geometry bound.Achieving error δ requires N ≍ δ^-d neurons absent a lower-dimensional structure.
  • Limitations: The exact framework does not cleanly cover multi-head attention concatenation and projection, causal masking, or several other practical architectural details.The exact Hopf–Cole correspondence is specified for LSE feedforward networks and a particular normalized attention mechanism.

M Neural Networks as KP Tau-Functions

The paper interprets LSE layers and related activations through KP tau-functions, heat/Hamilton–Jacobi representations, tropical limits, and Gibbs moments. It then derives distinct classification consequences for SiLU and GELU, including exact obstructions, Lipschitz behavior, convexity, and attribution properties.

  • KP tau-functions: LSE outputs are logarithms of KP tau-functions, while ε → 0 produces tropical tau-functions and max-type behavior.The full Toda interaction structure is not captured by the free tau-function used here.
  • SiLU: SiLU is an exact Gibbs moment, weight-gradient, ratio of heat solutions, and tangent Hamilton–Jacobi solution, but belongs to no scalar solution, gradient, or linear-heat class.Its classification is therefore tied to differentiation or moment structure rather than scalar log-sum-exp representation.
  • GELU: GELU is an exact Hopf–Cole heat variable and signed heat slice, converges to ReLU as ε → 0, and cannot be reproduced by any nonnegative datum or positive exponential family.The unique reproducing datum is ReLU − ε^2δ_0, and the defect vanishes in the distributional limit.
  • Activation classification: SiLU and GELU occupy complementary dictionary classes: SiLU has exponential-rational poles, whereas GELU is entire of order two and is not interchangeable with SiLU.The paper characterizes SiLU on the Hopf–Cole or large-deviation side and GELU on the heat or central-limit side.
  • Lipschitz and convexity consequences: Solution-class gates preserve the tropical Lipschitz constant and become convex or proximal under support restrictions, unlike SiLU and GELU.For supp ν ⊂ [0,∞), the gate is convex and nondecreasing; supp ν ⊂ [0,1] additionally gives a nonexpansive proximal operator.
  • Obstructions and dimensional extension: Tied moment-class gates can leave the solution-class hull, while the multivariate impossibility results follow by restricting elementwise activations to affine lines.The paper reports this escape for SiLU and GELU and extends the scalar obstructions to every dimension.
Loading 2605.28983v2…