Source-linked AI summary
Dynamical coarse-graining of neural-network training
Lin-Juan Wang, Da Li, Huiling Duan, Jianxiang Wang
TL;DR
The paper asks whether neural-network training can be described by evolution laws on a much lower-dimensional state space, and what conditions such laws must satisfy. It develops theorems around a learning operator and demonstrates a ReLU moment closure whose training-loss predictions agree with direct gradient-flow training.
Problem
It is an open question whether neural networks admit lower-dimensional evolution laws that predict subsequent training dynamics, and what general conditions such laws must satisfy.
Source: p. 1Method
The paper defines a learning operator and establishes theorems for its descent, exact closure, and dynamical coarse-graining.
Source: p. 2, Proof of Theorem 3 · p. 6Results
Approximately 10^-13 maximum output error is achieved by the moment closure, while frozen-NTK error increases during training; its training-loss predictions agree with direct 720-dimensional ReLU gradient flow within numerical accuracy.
Source: p. 4Takeaways & Limitations
Exact closure alone does not establish dimensionality reduction: a reduced state must remain small as parameter count grows and its evolution stable across width.
Source: p. 4No generic five-moment closure is asserted for smooth activations such as GELU or SiLU.
Source: p. 3
Abstract
from arXiv · showhide
Dynamical coarse-graining of neural networks gives macroscopic observables and enables training to be predicted, and controlled in a state space far smaller than the parameter space. In this paper, we first define the learning operator following the training law of neural networks and establish a descent theorem for the learning operator, which serves to determine whether the operator is sufficient to characterize the learning dynamics. Then the exact closure theorem and the dynamic coarse-graining theorem are presented in sequence. The three theorems established in this paper provide a systematic framework for constructing and assessing dynamical coarse-grained models of neural-network training. Finally, we demonstrate the applicability of the theorems with a finite-width nonlinear ReLU network. Its dynamic coarse-grained model for a fixed pattern is exactly derived and composed of five independent moments. The model exactly predict the training outputs and evolving learning operator. The framework may also guide the development of approximate coarse-grained models for more general neural networks when exact closure is unavailable.
Appendix: Proofs of the three theorems · Proof of Theorem 1
Theorem 1’s proof establishes that a learning operator descends through the coarse-graining map exactly when it is constant on each fiber. It also proves equivalent tests using vertical derivatives and projected velocities under stated conditions.
Proof of Theorem 1
- If K factors as K◦h, then K has the same value at any two microscopic states with the same coarse-grained state.Source: Proof of Theorem 1 · p. 5
- Fiberwise constancy makes the descended operator well-defined, and local smooth sections show it is smooth.Source: Proof of Theorem 1 · p. 5
- Surjectivity of h ensures the descended operator is unique, since every coarse-grained state has a microscopic preimage.Source: Proof of Theorem 1 · p. 5
- When h is a submersion, vanishing vertical derivative makes K constant along paths in each connected fiber.The proof integrates the derivative along piecewise C1 paths joining points in that fiber.Source: Proof of Theorem 1 · pp. 5–6
- Connectedness is necessary for this infinitesimal implication: vanishing tangential derivative constrains K only within individual fiber components.Source: Proof of Theorem 1 · p. 6
- For losses whose gradients span RN at each coarse-grained state, agreement of projected velocities across a fiber implies K is constant there.If K is already constant on fibers, the projected velocity −K(u)∇uℓ(u) is the same at every microscopic point over u.Source: Proof of Theorem 1 · p. 6
Proof of Theorem 2
The proof establishes that the learning operator factors through the coarse-graining map exactly when it is constant on each fiber, yielding a unique smooth reduced vector field. It then shows that microscopic trajectories project to reduced trajectories on their common existence interval.
Proof of Theorem 2
- Constancy of W(θ) = DΠ(θ)F(θ) on each fiber is equivalent to a unique smooth factorization W = G ◦ Π.Surjectivity of Π gives uniqueness, while smooth local sections define G smoothly and consistently across neighborhoods.Source: Proof of Theorem 2 · p. 6
- The vector factorization is equivalent to the r scalar equations DF φ_i = G_i ◦ Π, completing the stated equivalences.Source: Proof of Theorem 2 · p. 6
- By local ODE uniqueness, projecting a microscopic solution with Π gives the reduced solution from the corresponding initial state, whenever both solutions are defined.The argument assumes only a common existence interval, not global completeness or additional invariance.Source: Proof of Theorem 2 · p. 6
Proof of Theorem 3
The proof establishes trajectory convergence by comparing the finite-width and limiting dynamics, and shows that exact smooth state factorization constrains the state dimension for regular families.
Proof of Theorem 3(a)
- Integrating the finite-width and limiting dynamics and applying Grönwall’s inequality gives a uniform trajectory comparison under the stated common-region and Lipschitz assumptions.The vector fields must be uniformly within ε_m on a region containing both trajectories.Source: Proof of Theorem 3 · p. 6
Proof of Theorem 3(b)
- Exact smooth state factorization forces r_m/P_m ≥ c on any regular family whose rank is at least cP_m, ruling out asymptotically vanishing relative dimension there.The proof uses the chain rule for Ω_m = R_m ◦ bΠ_m to obtain the pointwise bound.Source: Proof of Theorem 3 · p. 6