Source-linked AI summary
A Function-Space Approach to the Statistical Mechanics of Learning Dynamics
Yizhou Zhang, Weichen Wu, Lun Du, Zhengjie Miao
TL;DR
Deep neural networks display regular macroscopic behavior despite nonlinear dynamics in enormous parameter spaces, motivating a theory that identifies an appropriate collective level. The paper formulates conditional statistical mechanics directly in function space, treating networks as microscopic realizations, and finds a thermodynamic preference for pairing faster relaxation with low-curvature, data-adaptive directions. This result is presented as one contribution to the slow dynamics rather than a complete dynamical theory.
Problem
The paper addresses how highly nonlinear learning dynamics can produce stable and scalable macroscopic organization.
Method
It formulates conditional statistical mechanics in function space, with parameter configurations as microscopic realizations and functions plus their dynamical operators as macroscopic variables.
Results
The conditional thermodynamic contribution favors pairing faster relaxation with the low-curvature, data-adaptive sector.
Takeaways & Limitations
Function space provides a natural macroscopic level for separating the function-space organizing principle from architecture- and parameterization-specific realizations.
Takeaways & Limitations
The identified preference is one thermodynamic contribution; realizing it in full training dynamics depends on remaining slow operator dynamics and parameter motion.
Abstract
from arXiv · showhide
Deep neural networks exhibit regular macroscopic behavior despite highly nonlinear dynamics in vast parameter spaces. We develop a statistical-mechanical description of learning directly in function space, treating parameter configurations as microscopic realizations and functions with their dynamical operators as macroscopic variables. For mean-squared loss, the exact error dynamics are governed by the learning operator \(M=JJ^\ast\). Combining the dynamical Boltzmann weight of the conditional stochastic dynamics with the parameter-space density of states, whose local curvature defines a statistical operator \(B\), and integrating over local fluctuations yields $$ Φ_{\mathrm{fluc}}(M;B)=\frac{σ_ξ^2}{2}\log\det(M^{-1}+B)+\mathrm{const}. $$ At fixed spectrum, this term is rotationally stationary when \([M,B]=0\), is minimized by pairing large eigenvalues of \(M\) with small eigenvalues of \(B\), and generates a local restoring contribution against rotational mismatch. For ReLU-type function spaces under mild stable statistical conditions, \(B=σ_ξ^2L^\ast\mathcal K L\), where \(L\) measures coarse-grained second-order structure. Thus the low-\(B\) sector corresponds, up to bounded anisotropy of \(\mathcal K\), to low structural curvature, implying a preference for faster relaxation along smooth, data-adaptive directions. These results identify function space as a natural macroscopic level for studying stable collective organization in learning.
1 Introduction
The paper develops a function-space statistical mechanics for explaining stable macroscopic organization in highly nonlinear neural-network learning. It separates dynamical geometry from parameterization-induced statistical geometry and derives a curvature-related preference for faster relaxation along structurally smooth directions.
- The paper asks how highly nonlinear learning dynamics produce stable and scalable macroscopic organization.
- For mean-squared loss, M describes learning dynamics while B describes statistical geometry induced by parameter microstate multiplicity.Only B compressed to the finite-dimensional active sector of M enters the determinants and commutators.
- For ReLU-type spaces, B is linked through L and K to coarse-grained second-order structure, so low-B directions correspond up to bounded anisotropy to low structural curvature.The resulting preference pairs faster relaxation with the low-curvature, data-adaptive sector.
- The framework identifies function space as a natural macroscopic level for separating learning dynamics from parameterization constraints.The paper presents this as one thermodynamic contribution rather than a complete account of the slow dynamics of M.
- It treats parameter configurations as microscopic realizations and functions with their evolution operators as macroscopic variables.The formulation is specified at the function-space level before selecting a concrete architecture.
- The conditional fluctuation free energy is rotationally stationary when [M, B] = 0 and is minimized by pairing large eigenvalues of M with small eigenvalues of B.Its matched state has positive rotational curvature and contributes a local restoring force against operator mismatch.
2 Related Work
The paper positions its contribution relative to kernel, stochastic-gradient, spectral-bias, and ReLU piecewise-linear approaches. It reverses the usual construction order by formulating conditional statistical mechanics in function space before asking how concrete architectures realize it.
- Unlike fixed-kernel descriptions, the framework formulates conditional statistical mechanics at the function-space level and then studies model-specific realizations.Architecture still determines which dynamical operators and microstate geometries are realizable.
- The stochastic-mechanical construction combines error-space dynamics with a parameter-space reference measure pushed forward to a function-space density of states.The resulting entropy reflects parameter-space multiplicity at fixed function-space macrostate rather than only the volume of a loss minimum.
- The framework differs from externally chosen Fourier or kernel decompositions by defining smoothness through data-weighted microstate geometry.Its structural bias is not imposed through a fixed Fourier basis or fixed kernel spectrum.
- For ReLU spaces, piecewise-affine geometry supplies a structural operator whose entropy curvature enters the thermodynamic preference.Under mild statistical boundary conditions, the corresponding microstate curvature is related to coarse-grained structural curvature.
3 Conditional Statistical Mechanics of Error Fluctuations
The paper formulates learning statistically in function space, combining exact error dynamics with parameter-induced microstate multiplicity. Conditioning on a macrostate yields a local fluctuation ensemble whose curvature defines the relevant statistical geometry.
- Function-space dynamics: Function-space error dynamics are governed exactly by the self-adjoint positive-semidefinite operator M, which may evolve during training.The construction does not require linearizing the network around a fixed parameter configuration.
- Interpretation: The bare conditional dynamics supplies the dynamical weight, while the pushforward reference measure counts parameter microstates realizing retained function-space states.The density is defined on a finite-dimensional retained sector rather than an assumed infinite-dimensional function-space volume.
- Microscopic and macroscopic descriptions: Parameter configurations provide microscopic realizations, while functions and their dynamical operators serve as macroscopic variables.The retained function sector preserves learning geometry while coarse-graining over redundant parameterizations.
- Conditional ensemble: The conditional ensemble factorizes the dynamical Boltzmann weight from the parameter-space density of states supplied by the reference microstate measure.This is an ensemble construction, not the stationary law generated by the bare conditional dynamics alone.
- Local fluctuation ensemble: The local fluctuation ensemble is Gaussian to quadratic order around the conditioned macrostate, with its Hessian determining the fluctuation geometry.The construction is conditional and does not require a time-scale separation or local equilibration along an actual training trajectory.
4 Operator Matching under Thermodynamic Stability
The conditional fluctuation free energy determines a precise orientation preference between the learning operator M and statistical curvature B. On fixed-spectrum rotations, stationarity requires commutation, while the minimum uses reverse spectral pairing and produces local restoring curvature.
- Conditional comparison: The fluctuation contribution is evaluated on fixed-rank, fixed-spectrum rotations while the conditioned macrostate and compressed statistical geometry remain fixed.This is a partial conditional comparison that excludes responses of r or B to parameter motion.
- Stationarity: Rotational stationarity holds if and only if M and B commute.At stationary configurations, the two operators share an eigenbasis.
- Reverse spectral pairing: The conditional orientational free energy is globally minimized by pairing larger eigenvalues of M with smaller eigenvalues of B.Same-order pairing is the global maximum, while other nondegenerate commuting permutations are saddles.
- Local restoring force: At the reverse-paired minimum, every nondegenerate pair has strictly positive quadratic curvature, yielding a local restoring force against rotational mismatch.Degenerate eigenvalue pairs remain flat directions for this contribution.
- Scope: Descent of the conditional fluctuation contribution locally reduces operator noncommutativity, but the full slow dynamics may contain additional terms.The result characterizes one thermodynamic contribution rather than the complete dynamics of M.
5 ReLU Cell Geometry and Structural Smoothness Preference
For ReLU function spaces, coarse-grained second-order structure is concentrated at activation boundaries and mapped into a positive semidefinite microstate-curvature operator. Under stable statistical conditions, its low-spectrum sector corresponds approximately to structurally smooth, data-adaptive directions that receive the faster-relaxation preference.
- ReLU cell geometry: ReLU functions are affine within activation cells, so their second-order structure is carried by activation boundaries and associated gradient jumps.The Hessian is therefore interpreted distributionally rather than as an ordinary smooth field.
- Structural representation: The structural map L sends retained functions to a coarse-grained field measuring departures from local affinity.Coarse-graining suppresses facet-scale singular structure before the local statistical description is applied.
- Microstate curvature: Under R1–R3, the retained-sector microstate-curvature operator is positive semidefinite, with its kernel equal to ker L under strict coercivity.R3 is an additional statistical stability assumption, not a consequence of ReLU geometry alone.
- Smoothness interpretation: Low-B_a spectral sectors correspond, up to the bounded anisotropy of K, to low coarse-grained structural curvature.Exact ordering of structural norms is not asserted when the structural metric is strongly anisotropic.
- Data-adaptive preference: The resulting preference pairs larger M eigenvalues with low-curvature, data-adaptive directions and therefore favors faster relaxation there.This conclusion concerns the conditional fluctuation contribution and depends on the remaining operator dynamics for realization during training.
6 Discussion and Future Work
The discussion extends the function-space framework to related learning regimes and identifies scope boundaries for interpreting its thermodynamic preferences. It connects possible macroscopic-state changes to grokking while emphasizing that branch transitions and nonequilibrium effects remain outside the present local theory.
- Learning regimes: The framework can describe fixed-operator kernel-like learning and slower operator adaptation that biases regular operator matching.It also allows sharp behavior to arise from competition between distinct macroscopic states.
- Grokking: Grokking is presented as a suggestive case where error-dependent statistical geometry could permit crossings between competing macroscopic branches.The proposed branch free energies would include macrostate, spectral, rank-dependent, and fluctuation contributions, but no global branch theory is constructed here.
- Scope: The training state may act as an endogenous control variable, while establishing branches, barriers, and transition dynamics is left for future work.The limitation applies to the local conditional theory developed in the paper.
- Limitations: The construction assumes isotropic function-space noise; a covariance geometry Q could alter reversibility and the resulting operator preference.Nonequilibrium extensions are identified as an important direction.
- Limitations: The fluctuation term is only one contribution to the slow dynamics, and interpreting its gradient as realized training dynamics requires other contributions not to overwhelm it.A quasi-static interpretation along training trajectories additionally requires local equilibration assumptions.
7 Conclusion
The conclusion presents function space as a macroscopic level for statistical mechanics of learning. The theory combines conditional dynamics with parameterization-induced multiplicity to derive an operator preference that, for ReLU-type spaces, favors faster relaxation in low-curvature data-adaptive sectors.
- Framework: Function space separates learning dynamics from statistical constraints induced by parameterization, with parameters as microscopic realizations and functions and evolution operators as macroscopic variables.This makes architecture a realization of the organizing principle rather than its starting point.
- Conditional ensemble: Integrating local error fluctuations in the conditional ensemble yields a free-energy contribution governed by the local statistical curvature operator B.The construction combines the exact mean-squared-loss error dynamics with parameter-space microstate multiplicity.
- Operator preference: At fixed spectrum, the fluctuation contribution is stationary for commuting M and B, minimized by reverse spectral pairing, and restoring against rotational mismatch.Reverse pairing matches larger M eigenvalues with smaller B eigenvalues.
- ReLU interpretation: For ReLU-type spaces under stable structural statistics, the active-sector spectrum of B measures coarse-grained structural curvature up to bounded metric anisotropy.The result depends on the stated structural conditions R1–R3.
- Implication: The resulting thermodynamic contribution favors pairing faster relaxation with the low-curvature, data-adaptive sector.The conclusion identifies this as a function-space organizing principle, not a complete account of slow dynamics.
A Residual-Preserving Rotations of the Local Conditional Free Energy
The appendix isolates how the fluctuation contribution governs residual-preserving rotations of the local conditional free energy. Under an additional common-invariant-mode condition, its reverse-pairing result becomes exact for the total quadratic local free energy on that restricted orbit.
- Setup: The appendix restricts attention to the fluctuation-induced orientational variation while holding the conditioned residual state and relevant geometry fixed.The fixed-spectrum orbit preserves eigenvalues, rank, and active subspace.
- Restricted rotations: Residual-preserving rotations are generated within the stabilizer of the active residual, and stationarity is characterized by the exact restricted condition without an invariant-subspace assumption.The appendix then examines a more transparent common-invariant-mode case.
- Invariant-mode case: When the residual is a common invariant mode of M and B, the orthogonal block is invariant under both operators and inherits the reverse-pairing result.The restricted minimum pairs eigenvalues in reverse order.
- Interpretation: Under the common-invariant-mode condition, reverse pairing exactly describes the total quadratic local free energy on the residual-preserving orbit.The main text does not require this extra condition and instead studies the fluctuation contribution on the full fixed-spectrum orbit.
- Zero-residual case: If the active residual vanishes, the macrostate term disappears and the fluctuation contribution supplies the complete quadratic orientational dependence.In that case the stabilizer is the full orthogonal group.