The Hamilton-Jacobi Theory of Deep Learning

• Training searches over Hamilton–Jacobi initial-value problems, with weights encoding initial data and inference evaluating the resulting PDE solution at x.The correspondence holds for arbitrary weights, so initialization already specifies one such equation; gradient descent selects the problem fitting the observations.Source: §1 · pp. 1–2
• ε continuously connects LSE layers to tropical max: ε →0 is an exact semiring homomorphism, while ε >0 yields the Hopf–Cole heat propagator of viscous Hamilton–Jacobi equations.The same ε is simultaneously softmax temperature, PDE viscosity, and convex regularization strength, as identified in Theorem 7.1.Source: §1 · p. 2
• Layerwise LSE networks exactly encode Hopf–Cole solutions, while composition corresponds structurally to PDE semigroups; attention, ResNets, RNNs, LSTMs, and SSMs inherit related correspondences.Transformer attention is an expected value under the same Gibbs measure, and recurrent or residual architectures discretize ODE characteristics with architecture-dependent viscosity.Source: §1 · p. 2
• The commutative diagram identifies one ε-indexed object across neural networks, tropical algebra, PDEs, and convex optimization, under Lipschitz g and convex H.The exact Hopf–Cole correspondence is complete for the quadratic Hamiltonian class, while broader architectures receive structural correspondences.Source: §1 · p. 2
• O(n^-1/(d+2)) is the minimax generalization rate achieved by ε*≍N^-1/d, alongside ε-controlled adversarial robustness and backpropagation as the residual-network co-state equation.The framework also gives an exactly 1 Lipschitz-inflation factor per layer and extends to learnable quadratic Hθ(p) = p⊤Aθp.Source: §1 · p. 2
• O(N) softmax influence weights πj provide closed-form attribution with exact label sensitivity, while attribution entropy undergoes fold bifurcations that annihilate attribution basins as ε increases.The label sensitivity is ∂f̂/∂gj = πj(1 + (f̂ − gj)/ε), requiring no Hessian inversion.Source: §1 · p. 2

Log-Sum-Exp and Convex Duality

• Log-sum-exp is smooth and convex for ε > 0, has softmax gradient, negative-entropy conjugate, and represents Gibbs, max-relaxation, and heat-equation structures.The Hopf–Cole substitution identifies log-sum-exp with a heat-equation solution operator.Source: §2 · p. 3

The Tropical Semiring

• The tropical semiring replaces addition with max and multiplication with addition, yielding tropical matrix multiplication as max-plus linear algebra [Litvinov, 2007].Its additive and multiplicative identities are −∞ and 0, respectively.Source: §2 · p. 3

Hamilton–Jacobi PDEs

• Viscous Hamilton–Jacobi equations use viscosity ε; Hopf–Cole linearizes the quadratic-Hamiltonian case, while convex Hamiltonians admit Hopf–Lax viscosity solutions [Hopf, 1950, Evans, 2010; Crandall and Lions, 1983; Lax, 1957].The inviscid equation corresponds to ε = 0, and general convex Hamiltonians produce a nonlinear transformed equation.Source: §2 · p. 3

Ultradiscretization

• Ultradiscretization maps smooth log-sum-exp systems to max-plus systems as ε → 0, while reverse lifting restores a positive-ε smooth system.The principle, originally developed for soliton PDEs, is applied here to neural networks.Source: §2 · p. 3

• LSEε approximates max within ε log m, and its ε → 0 limit realizes Maslov dequantization [Litvinov, 2007].For all x ∈ R^m, max_i x_i ≤ LSEε(x) ≤ max_i x_i + ε log m.Source: §3 · p. 3
• As ε → 0, the LSE arithmetic deforms into the tropical semiring, where addition becomes max and multiplication becomes +.For ε > 0, (R, LSEε, +) is isomorphic to (R≥0, +, ×), while the limit maps onto (R, max, +).Source: §3 · p. 3
• For ε > 0, affine-plus-LSE layers are smooth entropy-regularized optimizers with softmax contributions, whereas ε = 0 yields max-affine splines that partition inputs into polyhedral regions [Balestriero and Baraniuk, 2018, 2021].Softmax weights continuously relax one-hot region indicators, and the max-affine operator behaves like a decision-tree partition [Aytekin, 2022].Source: §3 · p. 3
• ReLU is recovered at ε = 0 through a two-neuron max comparison, while Softplus is the same operation at nonzero ε [Plate et al., 2026].The deformation therefore continuously interpolates between smooth activation and its tropical limit.Source: §3 · p. 3

LSE Representation

• An LSE layer exactly equals a Hopf–Cole solution of the viscous Hamilton–Jacobi equation under a discrete measure, rather than merely approximating it.The PDE residual is identically zero at fixed depth, so no physics-informed residual loss is required; training searches over initial conditions encoded by the parameters.Source: §4 · p. 4
• The network–PDE dictionary maps weights to support points, biases to transport costs, ε to viscosity, and width N to measure discretization.With W_j = y_j/(2t), each neuron is both a standard pre-activation and a kernel center computing a squared-distance interaction.Source: §4 · p. 4

The time parameter as gauge freedom

• The time parameter is gauge freedom: the same network map admits multiple self-consistent PDE readings, including data-scale, information, and generalization gauges.The generalization gauge sets ε*(t) = N^-1/d, while the information gauge maximizes attribution entropy.Source: §4 · p. 4

Transformer Attention

• L2 attention is an exact Hopf–Cole construction: its partition function equals the unnormalized solution, and its output is a value-weighted gradient of log-sum-exp.As ε approaches zero, softmax becomes hard argmax attention; no approximation is made for the L2 identity.Source: §4 · p. 5

Maximal Hamiltonian class

• The exact identity is restricted to quadratic Hamiltonians: anisotropic positive-definite matrices replace the isotropic quadratic form, while nonquadratic Hamiltonians leave a residual reaction term.The isotropic case requires H(p) = |p|^2; for general A ≻ 0, the spatial shift becomes x^T A^-1 x/(4t).Source: §4 · p. 5

Tropical limit

• As ε → 0, the Hopf–Cole solution converges to the Hopf–Lax formula, the unique Lipschitz viscosity solution of the inviscid HJ equation.The limiting equation is ∂_t u + |∇u|^2 = 0 with initial condition g.Source: §5 · p. 5
• In the discrete-measure setting, the tropical predictor is simultaneously a tropical inner product, MASO, linear program, and piecewise-affine function with support-point breakpoints.These equivalent descriptions connect the tropical limit to convex optimization and piecewise-affine network structure.Source: §5 · p. 6

ResNets and ODE Characteristics

• As h → 0 and L → ∞ with Lh = T fixed, ResNet Euler layers converge to ODE characteristics of an HJ PDE with Hamiltonian H(x,p) = p · F(x,θ).The construction applies to any smooth recurrence x_ℓ+1 = x_ℓ + hF(x_ℓ,W_ℓ); the discrete chain rule remains exact at finite h.Source: §5 · p. 6

Recurrent Architectures: RNNs, LSTMs, and SSMs

• RNNs, SSMs, and LSTMs fit the HJ-characteristic framework, with SSMs yielding quadratic Hamiltonians and max-plus limits, and LSTM gates becoming binary selectors tropically.RNNs arise from unit-step Euler discretization; SSM characteristics may be non-autonomous, while LSTM gates are gradients of two-neuron LSE layers.Source: §5 · p. 6
• Viscosity is global in feedforward networks, input-dependent in SSMs, and gated per cell in LSTMs, while LSEs remain closed under composition, affine transformations, and residual connections.This distinguishes how the deformation parameter enters across architectures while preserving the compositional framework.Source: §5 · p. 6

• ε indexes four equivalent perspectives: LSE/softmax versus max/ReLU, conventional versus tropical algebra, viscous versus inviscid Hamilton–Jacobi equations, and entropic regularization versus LP vertices.For neural networks, ε > 0 gives LSE/softmax and ε = 0 gives max/ReLU.Source: §6 · pp. 6–7
• Table 5 provides the full dictionary linking these four ε-dependent formulations.The dictionary is located in Appendix H.Source: §6 · p. 7
• Appendix E develops infinite-width and overparameterization consequences, including a Gaussian-process connection and double descent interpreted as near-shock formation.Source: §6 · p. 7

Commutativity theorem

• Under Lipschitz conditions, composing layers corresponds to the HJ semigroup through the tower property, while the network passes point evaluations and the PDE semigroup integrates the full function.The exact correspondence is established for the quadratic Hamiltonian H(p) = |p|^2 and its anisotropic extension, with the appropriate network form for general convex superlinear H.Source: §7 · p. 7

Limit relations

• As N → ∞, u_N^ε → u^ε at rate O(N^-1/d), while as ε → 0, u^ε → u^0 pointwise.The quadrature rate depends on ε through concentration of the softmax weights, motivating the Lipschitz assumptions.Source: §7 · p. 7
• The limits ε → 0 and N → ∞ can be exchanged: both orders applied to u_N^ε yield the same tropical limit u^0.This is the diagram’s nontrivial commutativity claim.Source: §7 · p. 7

Four perspectives

• At ε > 0, one LSE layer has four exact descriptions: neural forward pass, Gibbs log-partition function, viscous HJ solution, and entropy-regularized convex program.At ε = 0, each description has a corresponding tropical counterpart.Source: §7 · p. 7

Generalization and Robustness

• O(n^-1/(d+2)) is the fixed-t generalization rate, with optimal viscosity ε* ≍ N^-1/d and width N ≍ δ^-d for target error δ.The viscosity matches grid spacing, while depth-L composition adds a factor of L to the semigroup error; the rate includes an O(t) Moreau-envelope bias.Source: §8 · p. 8
• Increasing ε suppresses curvature and widens the Hessian-based certified radius, while ε → 0 produces shock formation and a near-shock double-descent peak at N/n → 1 [Emami et al., 2020].The unconditional ε-free mean-value certificate remains tighter; viscosity therefore exposes a robustness–expressiveness tradeoff rather than universally improving certification.Source: §8 · p. 8

Backpropagation as the Adjoint HJ Equation

• Backpropagation is reverse-time Euler integration of the Hamiltonian co-state equation, and weight updates execute the Pontryagin Maximum Principle for the network’s optimal initial-value problem.For LSE layers, the backward pass is exactly the adjoint heat semigroup; the initialization NTK is positive definite almost surely for N ≥ n, while stability follows from a CFL co-state bound.Source: §8 · pp. 8–9

Hallucination, Scaling, and Injection Attacks

• Outside the diffusion radius √(2εt) of every support point, outputs are exponentially close to dominant-neuron linear extrapolation, independently of the training distribution at x.Under intrinsic dimension deff, population loss scales as N^-1/deff and squared-error or cross-entropy loss as N^-2/deff; published estimates range from deff ≈ 5.3 to 26.3 [Kaplan et al., 2020, Henighan et al., 2020, Hoffmann et al., 2022].Source: §8 · p. 9

Attribution Landscape and Bifurcation

• As ε increases, attribution entropy undergoes fold bifurcations in which saddles annihilate adjacent minima, merging attribution basins; as ε → 0, minima occur at the support points.The entropy is the Gibbs-measure entropy at the network’s temperature, with gradients computable from the support-point Gibbs covariance.Source: §8 · pp. 9–10

Identity verification

• The core identity holds to approximately 10^-16, while the attention identity has exactly 0 error for d ∈ {4, ..., 64}.These results verify Theorem 4.1 numerically, as reported in Tables 2 and 3.Source: §9 · p. 10

Quadrature scaling

• ℓ∞ error decays as O(N^-1/d) for Lipschitz g(y) = |y| at fixed (ε, t), confirming the predicted quadrature rate in the well-resolved regime.The gauge-coupled headline rate is established analytically in Appendix E.Source: §9 · p. 10

Generalization scaling

• At optimal ε* = N^-1/d_eff, RMSE scales as N^-α with α = 1/d_eff, while fitted α̂ > 1/d_eff shows smoother g outperforms the minimax Lipschitz bound.Figures 5 and 6 verify Proposition 8.8 using both closed-form and Adam-trained procedures.Source: §9 · p. 10

Real-data validation

• Corollary 8.2 is verified numerically and extended from synthetic settings to MNIST and CIFAR-10.The real-data extension is shown in Figure 8.Source: §9 · p. 10

10 Conclusion

• Theorem 7.1 closes the network–tropical–PDE–optimization correspondence into a commutative diagram governed by ε, with trained weights encoding the Hamilton–Jacobi equation evaluated by the network.The same framework yields generalization, robustness, hallucination geometry, scaling, and Hamiltonian extensions, while nearest-neighbor retrieval, convex regularization, and entropic optimal transport appear as limiting cases.Source: §10 · p. 10

A The Physical Analogue: Neural Networks as Path Integrals

• The LSE forward pass is a discrete path integral and imaginary-time Schrödinger propagator, with neuron paths weighted by Gibbs factors e^(-cost_j/ε).The positive Gibbs measure makes the representation classically computable, while ε → 0 concentrates it on the cost-minimizing neuron and recovers the classical tropical limit.Source: A The Physical Analogue: Neural… · p. 15
• ResNets implement cellular-automaton-like iterations of Hamilton–Jacobi characteristic steps, with depth L and step size h serving as discrete time and lattice spacing.The forward recurrence computes HJ characteristics and the backward pass computes their co-state equations, sharpening the ODE and SDE interpretations of residual networks.Source: A The Physical Analogue: Neural… · p. 15, B Related Work · p. 17
• The LSE network at ε > 0 is a universal classical simulator for HJ initial-value problems with optimal O(N^-1/d) approximation for Lipschitz data, using positive Gibbs measures rather than quantum hardware.This contrasts with quantum simulation, where negative Wigner-function values obstruct efficient classical simulation.Source: A The Physical Analogue: Neural… · pp. 15–16

B Related Work

• The framework identifies ReLU and piecewise-linear spline theories as the ε = 0 tropical limit, while viscous solutions converge uniquely to viscosity solutions and soft attention converges to hard attention.It credits foundational HJ, tropical, spline, and viscosity results while positioning the ε → 0 limit as their common structure.Source: B Related Work · p. 17
• The HJ formulation provides a mechanistic account of scaling from intrinsic data dimension, finite-width quadrature bounds, adaptive Gibbs-based feature selection, and an exact LSE neural tangent kernel.It also relates ε to spectral scale and connects the framework to cross-architecture scaling consistency, flow matching, score diffusion, and optimal transport.Source: B Related Work · pp. 17–20
• Unlike prior HJ-solving architectures, this work treats the log-sum-exp form as the exact computation of trained deep networks and adds commuting width–tropical limits, compositional error bounds, and activation classifications.It distinguishes these foundational results from consequences including reconstruction, robustness, attribution, and scaling results.Source: B Related Work · p. 18
• The framework recasts Moreau–Yosida smoothing, entropic optimal transport, modern Hopfield retrieval, and Pontryagin-based ResNet analyses as aspects of the same HJ structure.In particular, retrieval evaluates the HJ solution at a query, while the forward–backward residual system is an exact Hamiltonian flow rather than only an optimal-control analogy.Source: B Related Work · pp. 19–20

Architecture as PDE, Output as Solution

• Log-sum-exp layers exactly encode viscous Hamilton–Jacobi solutions, with weights representing initial data and zero PDE residual before training.Theorem 4.1 guarantees the identity for arbitrary weights, without residual loss or data assumptions.Source: Architecture as PDE, Output as Solution · p. 21, Regularity scope of Theorem 7.1 · p. 23
• Gradient descent searches over discrete initial conditions, selecting the viscous Hamilton–Jacobi equation whose propagated solution best fits observations.The architecture fixes the PDE class through its Hamiltonian, while optimization selects the initial data encoded by weights.Source: Architecture as PDE, Output as Solution · p. 21, Tasks as Initial-Value Problems · p. 21

Tasks as Initial-Value Problems

• Regression and kernel estimation are Hopf–Cole representations, while encoder–decoder networks compose propagators and interpret the bottleneck as an intermediate PDE state.For Gaussian kernels, ε = σ^2/2 and u_ε = −ε log v; encoder and decoder propagators compose into S_t1+t2.Source: Tasks as Initial-Value Problems · pp. 21–22
• Across tasks, outputs remain PDE solution values: classification uses softmax Gibbs measures, time series use ODE trajectories, and transformers apply HJ propagation across depth.Classification confidence is controlled by ε, while transformer depth acts as PDE time; the tropical limit sharpens decisions [Aytekin, 2022] and contextual embeddings arise after propagation [Tong et al., 2025].Source: Tasks as Initial-Value Problems · pp. 21–22

Regularity scope of Theorem 7.1

• Exact network–PDE identities and semigroup composition require no regularity, whereas quadrature convergence requires Lipschitz g and vanishing-viscosity convergence requires convex, superlinear H.The joint limits N →∞ and ε →0 commute only when both Lipschitz initial data and convex superlinear Hamiltonians provide uniform control.Source: Regularity scope of Theorem 7.1 · p. 23

SGD as initial-condition optimization in the mean-field limit

• In the mean-field limit, SGD becomes Wasserstein gradient flow over initial-data measures, identifying optimization with search for the risk-minimizing viscous HJ initial-value problem [Mei et al., 2018].For two-layer networks with 1/N normalization, finite-width particle error is O(N^-1/d), matching the PDE approximation rate; deeper extensions require additional propagation-of-chaos arguments.Source: SGD as initial-condition optimization… · pp. 23–24

Training dynamics for fixed-support LSE networks

• For fixed support and convex non-decreasing losses, the bias-parameter risk is convex, so every local minimum is global.The non-decreasing condition is essential; squared loss is generally non-convex in the biases, with MSE treated separately through NTK analysis.Source: Training dynamics for fixed-support… · p. 24
• With distinct inputs and absolutely continuous random supports, K(0) ≻ 0 almost surely, and kernel-linearized gradient descent reaches zero MSE linearly when η < n/λmax(K(0)).The same convexity and NTK conclusions apply to last-layer training with frozen lower representations.Source: Training dynamics for fixed-support… · p. 25

Proof of Theorem 4.1 (NN–PDE Identity)

• The NN–PDE identity is algebraically exact, with W_j = y_j/(2t) uniquely exposing the quadratic transport cost in each neuron’s Hopf–Cole exponent.No approximation is made; completing the square yields the exact correspondence between linear weights, biases, and the Hopf–Cole representation.Source: Proof of Theorem 4.1 (NN–PDE Identity) · p. 26

Proof of Theorem 5.1 (Tropical Limit via Varadhan’s Lemma)

• As ε →0, the viscous log-sum-exp solution converges to the Hopf–Lax infimum, turning soft memory weights into nearest-neighbor retrieval; the same limit holds for discrete measures.Varadhan’s lemma identifies the limit as the large-deviation rate function, while the discrete analogue becomes a minimum over atoms.Source: Proof of Theorem 5.1 (Tropical Limit… · pp. 26–27

Proof of Theorem 7.1 (Commutative Diagram)

• Deep composition follows the heat-semigroup Chapman–Kolmogorov law, and the continuous and discrete limits commute under Lipschitz conditions with an ε-free fixed-time discretization bound.The semigroup identity composes layer times additively; both orders of N →∞ and ε →0 converge to the inviscid solution.Source: Proof of Theorem 7.1 (Commutative… · p. 27

Multilayer Composition: Finite-Depth Error and Joint-Limit Exactness

• Finite-depth composition error grows linearly with depth at the single-layer O(N^−1/d) rate, while L →∞ with t_l = T/L and L^2 max_l N_l^−1/d →0 yields exact HJ-semigroup convergence.The finite-depth estimate uses non-expansive Lipschitz propagation; the joint-limit condition accounts for the 1/t_l growth of per-layer constants.Source: Multilayer Composition: Finite-Depth… · p. 28

Proof of Theorem 8.1 (Generalization)

• The fixed-t minimax generalization rate is O(n^−1/(d+2)), while the theorem’s resolution gauge gives approximation and discretization errors O(N^−1/d) and viscosity bias O(ε) after bias cancellation.Treating t as fixed keeps the weight bound fixed and recovers the minimax rate; the general comparison-principle alternative is O(√ε).Source: Proof of Theorem 8.1 (Generalization) · p. 29

Proof of Corollary 8.2 (Robustness)

• For LSE predictors, adversarial sensitivity is controlled by softmax-weighted weight norms and ε, with the Hopf–Cole predictor adding a quadratic 1/(4t) contribution that dominates at small t.The gradient is a convex combination of neuron rows, and the resulting Taylor bound yields an explicit ε-dependent robustness estimate.Source: Proof of Corollary 8.2 (Robustness) · p. 29

Proof of Theorem 8.5 (Backpropagation)

• Residual-network backpropagation is exactly the Pontryagin co-state equation, with forward Euler state dynamics and backward adjoint dynamics agreeing up to O(h^2); stability requires h∥∇_xF∥_2 < 2.For an LSE block, the stability threshold is explicitly tied to ε and the weight norm through h* ≍ 2ε/∥W∥.Source: Proof of Theorem 8.5 (Backpropagation) · p. 30

Proof of Proposition 8.6 (Hallucination) and Proof of Proposition 8.8 (Scaling Laws)

• The remaining structural consequences identify OOD extrapolation as an architecture-level hallucination regime and refine the approximation rate from O(N^−1/d) to O(N^−1/d_eff) when support lies on a d_eff-dimensional manifold.The hallucination criterion depends on the gap Δ(x)/ε, while manifold covering numbers determine the intrinsic-dimension scaling.Source: Proof of Proposition 8.6 (Hallucination) · p. 30, Proof of Proposition 8.8 (Scaling Laws) · p. 30

Proof of Proposition 8.8 (Scaling Laws)

• The scaling proof yields L ≍ n^-1/(d_eff+2), compute-optimal L(C) ≍ C^-1/(2d_eff+2), and cross-entropy width rate L_CE(N) ≲ N^-2/d_eff.The width-only prediction rate is O(N^-1/d_eff) for L1-type loss, while squared-error or cross-entropy losses square the deviation rate.Source: Proof of Proposition 8.8 (Scaling Laws) · pp. 30–31

Proof of Proposition E.6 (Measure Poisoning)

• For poisoned neurons, softplus bounds output damage by max(z0 − f_N^ε(x), 0) + ε log 2, while attacks exceed ε log 2 exactly when z0 − f_N^ε(x) > 0.When the gap is γ > 0, the shift tends to γ as ε →0; nonpositive gaps remain bounded by ε log 2.Source: Proof of Proposition 8.8 (Scaling Laws) · p. 31

Proof of Proposition 5.4 (Recurrent Architectures)

• RNNs, SSMs, and LSTMs are Hamiltonian characteristic discretizations, with LSTM gates formed from LSE gradients and becoming max-plus selectors as ε →0.RNNs use unit-step Euler dynamics; SSMs recover state and adjoint equations; LSTM forget, input, output, and cell gates compose two-neuron LSE-layer gradients.Source: Proof of Proposition 5.4 (Recurrent… · p. 31

Feynman–Kac Representation and the Gaussian Fixed Point

• The matched Gaussian scale is q* = 2εt, simultaneously the Brownian variance, Hopf–Cole kernel-matched prior scale, and one-time-unit heat-semigroup diffusion variance.At this scale, the Gibbs posterior has mean x/2.Source: Feynman–Kac Representation and the… · p. 32

Double Descent as Near-Shock Formation: Formal Analysis

• At N = n, curvature peaks at near-shocks, while above interpolation it decreases as (N/n)^-2/d, smoothing predictions and reducing test error.The near-shock boundaries become dense between training points, whereas overparameterization distributes them more finely and lowers their intensity.Source: Double Descent as Near-Shock… · p. 33

Variable-Hamiltonian Extension

• The variable-Hamiltonian extension replaces Euclidean transport with A_θ, improving quadrature to O(N^-1/r) when data is r-dimensional in the A_θ^-1 metric.Learning A_θ simultaneously adapts transport, reduces effective dimension, and smooths the Hopf–Cole solution through smaller weight norms.Source: Variable-Hamiltonian Extension · p. 34

Propositions

• The Hopf–Cole predictor is an exact label convex combination with attribution weights π_j, whose label sensitivity is computable in O(N) without second-order information.These weights explain a fixed input and differ from classical influence functions based on training-example reweighting.Source: Propositions · p. 35
• For sufficiently small ε, each support point y_k is a local minimum of the entropy landscape because attribution concentrates there and moving within its Voronoi cell increases entropy.The entropy gradient vanishes at y_k as cross terms decay exponentially, while smoothness and the implicit function theorem govern critical-point continuation before bifurcation.Source: Propositions · p. 36
• At each fold, a saddle coalesces with a local minimum, separating attribution basins below the threshold and merging them above it into hierarchical clustering.The crossing direction switches the dominant attribution, while the saddle entropy H(x*; ε) quantifies uncertainty at the transition.Source: Propositions · p. 36

Empirical Evidence

• Figure 1 shows the critical-point count falling from 9 to 1 through discrete fold bifurcations as ε increases in the synthetic two-cluster experiment.For N = 16, d = 2, and ε*≈0.25, saddle Hessian eigenvalues approach zero at each bifurcation, confirming branch annihilation.Source: Empirical Evidence · p. 36, Figure 1 · p. 37
• Figures 2 and 3 show the particle-to-wave phase transition: sharp low-viscosity attribution basins flatten toward uniform entropy, with ε*≈0.25 for synthetic data and ε*≈0.091 for MNIST 3-versus-7.On MNIST, the exact HJ decision boundary follows inter-cluster geometry in PCA space, while feature sensitivity concentrates along the digit-separation boundary.Source: Figure 2 · p. 38, Figure 3 · p. 39

Synthetic experiments

• The recovery, scaling, and robustness experiments use Adam with the stated step counts and learning rates, varying width, dimension, and viscosity ε on synthetic targets.Experiment A uses 10 neurons and 8 seeds; B uses 4,000/800 train/test samples and dimensions 1, 2, and 4; C uses 30 neurons and 3 seeds.Source: G Numerical Details · p. 40

Phase diagram and bifurcation

• The phase-diagram and bifurcation analyses are analytical sweeps without training, using 60 ε-frames for clustered supports and 200 ε-values for entropy bifurcations.Experiment H uses N = 16 points in d = 2 with ε* = 0.25.Source: G Numerical Details · p. 40

Quadrature convergence

• At fixed (ε, t) = (1, 1), quadrature errors follow O(N^-1/d) slopes across d ∈ {1, 2, 4}, confirming Theorem 8.1 step (i).This check uses the well-resolved regime where grid spacing h is finer than the Gibbs concentration scale 2εt.Source: G Numerical Details · p. 40, Figure 4 · p. 41

Scaling law

• Fitted exponents α ≈ 1/d confirm Proposition 8.8 for Adam-trained LSE networks across d ∈ {1, 2, 4}.Figure 6 reports test RMSE scaling with width N for target g(y) = ∥y∥.Source: G Numerical Details · p. 40, Figure 6 · p. 42

Hessian bounds

• The Hessian bound is never violated across ε ∈ [0.08, 1.0] for SGD-trained networks and across ε ∈ {0.1, 0.3, 1.0, 3.0, 10.0} on MNIST and CIFAR-10.The image experiments use N = 128 neurons with 50 or 64 PCA features, respectively.Source: G Numerical Details · p. 40

Analytical identities

• Transformer and attention identities are verified analytically to floating-point precision or exact zero error over 500 random trials per dimension.The checks use sequence length 8, dv = 16, and confirm attention plus LayerNorm equals ∇LSE and the LSE-FFN Hamilton-Jacobi identity.Source: G Numerical Details · p. 40, Table 4 · p. 43, Table 3 · p. 43

Universality

• Any continuous function on a compact domain is uniformly approximable by a deep LSE network, and equivalently by a discrete-measure viscous HJ solution, with depth at least two for non-convex targets.The approximation error can be made below any δ > 0; single LSE layers are convex, so depth ≥2 is required for general non-convex functions.Source: H Scope of the Correspondence:… · p. 44

Closure

• The HJ class is closed under LSE composition and affine input transformations exactly, while residual connections preserve it up to O(h) Euler discretization error.Each LSE layer is an exact Hopf–Cole solution for its own initial datum, and residual paths discretize HJ characteristics.Source: H Scope of the Correspondence:… · p. 44

Broader PDE classes

• For convex Hamiltonians, the correspondence extends to scalar conservation laws, eikonal equations, and linear transport, with weights encoding the Hamiltonian and ε encoding viscosity.Non-convex Hamiltonian analogues are suggested by ultradiscretized KdV and Toda hierarchies, but the neural-network construction remains open.Source: H Scope of the Correspondence:… · p. 45

Non-convex Hamiltonians

• Exact Hopf–Cole linearization holds precisely for quadratic Hamiltonians, whereas general non-quadratic Hamiltonians introduce a reaction term and invalidate the exact algebraic identity.The anisotropic-quadratic class H(p) = p^T A p is necessary and sufficient for the reaction term to vanish.Source: H Scope of the Correspondence:… · p. 45
• For non-convex H, the kernel network is exact for the convex envelope H** and approximates the original solution with error O(t · ||H − H**||∞).The error vanishes when H is convex; depth expands the representable function class but cannot reduce this intrinsic convexification gap.Source: H Scope of the Correspondence:… · pp. 45–46

Heat-kernel quadrature

• LSE networks act as heat-kernel quadratures: N →∞ recovers the Green’s-function integral, while ε →0 yields Hopf–Lax minimization and nearest-neighbor or Voronoi selection.The learned biases encode sampled initial data, and the tropical limit concentrates on the minimizing support point.Source: H Scope of the Correspondence:… · p. 47

Convolutional architectures

• Convolutional LSE layers solve the same HJ equation at every receptive field, with translation-equivariant parameter sharing corresponding to globally conserved co-state gradients.Position-independence of the Hamiltonian gives ṗ = 0 along characteristics, matching a shared filter across spatial locations.Source: H Scope of the Correspondence:… · p. 47

General activations and transformers

• SiLU is an exact Gibbs moment and GELU an exact heat flow of signed data; both share the LSE tropical limit, with σ(x/ε) and Φ(x/ε) converging to Heaviside(x).Replacing GELU or SiLU with LSE in transformer FFNs activates the exact HJ correspondence, while attention remains exact and residuals contribute O(h) error.Source: H Scope of the Correspondence:… · pp. 47–48

Efficient token insertion

• O(1) token insertion uses only the new pre-activation and cached output, via the measure-extension identity of Proposition E.6.No recomputation over previous tokens is required.Source: H Scope of the Correspondence:… · p. 49

Attention sinks

• Attention sinks are dominant neurons: attended values approach v_js with error at most (N −1)e^−∆_s(q_i)/ε max_j ||v_j −v_js||_2.The Gibbs measure concentrates exponentially on the minimum-cost sink, yielding the tropical Hopf–Cole limit.Source: H Scope of the Correspondence:… · p. 49

Positional encodings

• Additive positional encodings shift logits into query- and position-dependent initial data, while RoPE preserves the identity in rotated coordinates with unchanged g(y_j).For additive encodings, the shifted datum is g(y_j) 7→ g(y_j) − εφ(i,j); the Hopf–Cole identity remains valid for each fixed query by Theorem 4.1.Source: H Scope of the Correspondence:… · p. 49

Intrinsic dimension from scaling curves

• The Chinchilla scaling curve yields deff ≈5.7, while Kaplan’s larger apparent dimension reflects under-training from insufficient data.Compute-optimal allocation makes the Chinchilla estimate more interpretable as a geometric quantity.Source: I Intrinsic Dimension from Published… · p. 50, Table 7 · p. 50
• deff = 2/α converts published model-size scaling exponents into order-of-magnitude intrinsic-dimension estimates under squared-error or cross-entropy loss.The conversion follows Proposition 8.8, but empirical α conflates approximation, optimization, and data-coverage effects.Source: Table 7 · p. 50, I Intrinsic Dimension from Published… · p. 50
• Intrinsic dimension varies by domain: mathematical problem solving is lower-dimensional than language, while video is intermediate.This ordering is consistent with the manifold hypothesis, reflecting differences in structure and degrees of freedom.Source: I Intrinsic Dimension from Published… · p. 50

Optimal temperature selection

• ε* ≍ N^-1/d minimizes excess risk O(N^-1/d + N/n), so optimal temperature follows from width and intrinsic dimension without grid search.Temperatures below ε* let estimation error dominate, while higher values over-smooth through approximation error.Source: J Actionable Design Principles · p. 51

Robustness control

• Increasing ε monotonically reduces input sensitivity, providing a certified robustness guarantee without retraining; decreasing ε toward ε* sharpens discrimination.The worst-case sensitivity is bounded by ∥W∥2,∞/ε uniformly over inputs.Source: J Actionable Design Principles · p. 51

Temperature annealing

• Annealing ε from a high value toward ε* moves training from a smooth, diffusion-dominated regime toward sharper approximation near optimal viscosity.This schedule is the neural-network analogue of PDE continuation and jointly tracks conditioning, approximation quality, and robustness.Source: J Actionable Design Principles · p. 51

Architecture as discretization strategy

• Architecture choice is a discretization choice: feedforward networks sample measures, ResNets discretize characteristics, transformers perform Hopf–Cole attention averaging, and recurrent models discretize characteristics.The corresponding mechanisms are iid neurons, Euler–Maruyama dynamics, Gibbs attention at ε = √d_k, and architecture-dependent viscosity.Source: J Actionable Design Principles · p. 51
• Process-discretization architectures suit temporal tasks, measure-discretization suits static-distribution invariants, and attention suits query-dependent weighting of stored data.This maps task structure directly to the relevant Hamilton–Jacobi discretization strategy.Source: J Actionable Design Principles · p. 51

Backpropagation as adjoint optimization

• Backpropagation is the Hamiltonian system’s co-state equation, enabling symplectic integrators and higher-order ODE solvers as theoretically motivated alternatives for long-horizon training stability.The result identifies reverse-mode autodifferentiation with the adjoint method for PDE-constrained optimization.Source: J Actionable Design Principles · p. 51

Compute-optimal scaling forecasts

• α ≥ 2/d_eff and d_eff = 2/α connect scaling curves to intrinsic dimension, while N ∝ n^d/(d+2) forecasts compute-optimal allocation before large training runs.The framework supports estimating d_eff from published scaling curves or a small pilot run.Source: J Actionable Design Principles · p. 51

K Implications for Large-Scale Training, Alignment, and Continual Learning

• At fixed data size, excess risk hits an estimation floor, making width and intrinsic dimension the dominant levers once n ≫ N^(deff+2)/deff.Approximation improves as N^-1/deff, while architectural bias, curation, specialization, and sparse routing reduce deff; at deff = 26.3, halving error costs roughly 8.3 × 10^7 more neurons.Source: K Implications for Large-Scale… · p. 52
• Alignment requires training-support coverage: outside support, when Δ(x)/ε ≫ log N, outputs follow dominant-neuron extrapolation and cannot be governed by any loss function.The proposed interventions are broader data coverage, increased ε to smooth OOD extrapolation, and penalties at synthetic OOD points.Source: K Implications for Large-Scale… · pp. 52–53
• Continual learning preserves prior-task accuracy by adding capacity rather than displacing neurons: reallocating k of N neurons changes the bound from O(N^-1/deff) to O((N-k)^-1/deff).Freezing old neurons and generative replay both preserve prior support points in the encoded measure.Source: K Implications for Large-Scale… · p. 53

Limitations

• The framework does not characterize practical deep-network SGD selection: its precise training results are limited to two-layer mean-field or kernel-linearization regimes, leaving converged initial-data selection open.The mean-field result concerns Wasserstein gradient flow under Mei–Montanari conditions, while the finite-width kernel is Kab = ε2⟨π(xa), π(xb)⟩.Source: Limitations · p. 53
• Exact Hopf–Cole identities are maximal for quadratic Hamiltonians; non-quadratic cases receive convex-envelope bounds or structural correspondences, and activation identities beyond this class remain open.Under quadratic H, product-form GELU and SiLU retain exact identities in adjacent moment and measure classes.Source: Limitations · p. 53
• The minimax rate O(N^-1/d) still imposes the curse of dimensionality, while replacing d with deff depends on an empirical scaling estimate rather than certified support geometry.Achieving error δ requires N ≍ δ^-d neurons unless concentration near a lower-dimensional submanifold is justified.Source: Limitations · p. 53
• The LSE partition function occupies only the free-soliton sector of the KP hierarchy, lacking Hirota phase shifts; whether training learns genuine soliton scattering remains open.The framework instead unifies existing views through ε, which simultaneously represents softmax temperature, PDE viscosity, and proximal regularization [Litvinov, 2007].Source: Limitations · p. 54
• The exact architectural scope excludes clean formulations for multi-head concatenation, causal masking, ReLU or linear attention, and fully coupled attention readouts.Elementwise GELU and SiLU remain exact under quadratic H, but a uniform bound over all weight and value matrices is still open.Source: Limitations · p. 54

M Neural Networks as KP Tau-Functions

• LSE layers are logarithms of KP tau-functions at specific flow times, while ε → 0 produces the tropical tau-function associated with the Box–Ball System.The construction uses the free, non-interacting N-soliton sector; incorporating Toda pairwise interactions and depth-indexed compositions remains open.Source: M Neural Networks as KP Tau-Functions · p. 55

N Exact HJ identities for the product-form activations SiLU and GELU

• SiLU and GELU are exact product-form objects, not scalar log-sum-exp layers: their odd parts equal x/2 at every ε, preserving the ReLU reflection identity exactly.SiLU is a Gibbs moment and weight-gradient, whereas GELU belongs to complementary heat/measure constructions.Source: N Exact HJ identities for the… · p. 55

SiLU: the moment/tangent class

• SiLUε(x) is exactly a Gibbs moment, a ratio of heat solutions, and a tangent HJ solution, but belongs to none of the scalar solution, gradient, or linear-heat classes.Its weight-gradient interpretation identifies SiLU as the tangent of the solution class along the weight direction.Source: SiLU: the moment/tangent class · pp. 56–57

GELU: the measure class

• GELU is a Hopf–Cole heat variable and an exact signed heat slice with datum ReLU − ε^2δ_0, converging uniformly to ReLU at rate O(ε), but admits no positive Gibbs representation.Thus SiLU and GELU occupy complementary, non-interchangeable sides of the dictionary: tangent/moment versus measure.Source: GELU: the measure class · p. 57

Lipschitz and convexity consequences of the classification

• Per-layer Lipschitz inflation is 1 for solution-class gates, at least 1.0998 for SiLU, and 1.129 for GELU, compounding at depth L as 1.0998^L versus 1.129^L versus 1.Solution-class gates with nonnegative support are convex, and support in [0,1] additionally gives nonexpansiveness; SiLU and GELU satisfy neither property.Source: Lipschitz and convexity consequences… · pp. 58–59

Multivariate obstruction via line restriction

• Scalar impossibility results extend to every dimension because the relevant activation classes are closed under restriction to any affine line.For multivariate positive exponential representations, line restriction produces a one-dimensional positive exponential family via pushforward measure.Source: Multivariate obstruction via line… · p. 59
• SiLU and GELU components lie in neither multivariate solution class, and SiLU additionally lies outside the heat class, for every dimension and coordinate.The proof restricts each component to a coordinate line, reducing the claim to the scalar obstructions; SiLU’s pole also contradicts heat-class entirety.Source: Multivariate obstruction via line… · p. 59
• The obstruction extends coordinatewise to all dimensions, while generic coupled attention readouts inherit a SiLU-type obstruction along lines without a uniform bound over V and W.The attention claim applies to profiles of ⟨V, ∇LSEε(Wz)⟩ and is stated only generically.Source: Multivariate obstruction via line… · p. 59
• At fixed gauge t = ε/2, GELUε converges uniformly to ReLU as ε → 0, and Lipschitz-layer telescoping closes the network–tropical value-level edge at fixed depth.The gradient edges remain excluded, as in the softmax-to-arg-max limit.Source: Multivariate obstruction via line… · p. 59
• GELU’s PDE interpretation uses a signed, ε-dependent datum rather than a fixed Hopf–Cole initial-value problem, so it reaches the tropical corner through mollifier shrinkage instead of LSE dequantization.The relevant semigroup is still in the viscous family, but GELU has no solution-vertex object in the stated diagram.Source: Multivariate obstruction via line… · pp. 59–60
• GELU is not a semigroup factor because it is negative for x < 0, but uniformly Lipschitz interleaving with re-entry into downstream LSE layers preserves function-level composition and HJ semantics.Thus no obstruction is found at the locked gauge, although full four-vertex functorial closure is not claimed.Source: Multivariate obstruction via line… · pp. 59–60