Stable Neural Flows

I. INTRODUCTION

• Neural ODEs define the input–output map through an ODE flow ẋ(s)=f(s,x(s)) with initial state x(0)=u over a compact depth domain.The vector field is parametrized by neural-network weights, while the paper restricts its analysis to depth-invariant weights.Source: §I · p. 1, A. Preliminaries · p. 2
• Unconstrained neural ODEs may produce unstable or stiff dynamics, increasing sensitivity to input perturbations and numerical-solver overhead while permitting chaotic behavior.These effects can make generalization to unseen data unreliable because errors propagate from small feature deviations.Source: §I · p. 1

II. STABLE NEURAL FLOWS

• Stable neural flows replace unconstrained vector fields with neural-network-parametrized energy functionals whose negative gradients steer the trajectories.The energy-based construction is introduced as a stable neural ODE variant with trajectories evolving along energy level sets.Source: §I · p. 1, §II · p. 2

A. Preliminaries

• The framework treats parameter learning as optimization of a smooth data-fitting cost over model parameters, with input and output projections mapping data into and out of the latent state.The neural ODE receives u_t, evolves the state over the depth domain, and produces the estimate ŷ_t through h_y.Source: A. Preliminaries · p. 2
• Under Lipschitz vector fields, each input has a unique solution over the full depth domain, yielding a parameterized mapping from weights and inputs to trajectories and outputs.The resulting trajectory map is denoted φ_s(w,u_t), and the output can be written as ŷ_t=γ(w,u_t).Source: A. Preliminaries · p. 2

• Stable neural flows use a bounded, neural-network-parametrized energy whose negative gradient steers the state dynamics.The energy may depend explicitly on each input, producing a data-dependent family of energy functionals.Source: B. Stable Neural Flows · p. 2
• The model’s solutions are forward complete on any compact depth domain, and suitable parameters can reach an arbitrary level of steady state.The steady state is understood as closeness to an invariant set.Source: B. Stable Neural Flows · p. 2
• Under the proposition’s neighborhood and derivative conditions, closed sets satisfying the energy-minimum condition are locally asymptotically stable.The proof uses the energy as a Lyapunov function with nonpositive derivative along trajectories.Source: B. Stable Neural Flows · p. 2
• Almost all initial conditions converge toward a stable set associated with an isolated set of local energy minima, excluding local maxima and saddle-point basins.Each isolated set of local minima has a basin of attraction.Source: B. Stable Neural Flows · p. 2

1) Port–Hamiltonian inspired model:

• The port–Hamiltonian-inspired variant preserves stability when A(x)+A⊤(x) ≺ 0, while allowing combinations of negative gradient directions.This gives the model more freedom to steer states toward energy minima.Source: 1) Port–Hamiltonian inspired model · p. 2

2) Second–order model:

• The second-order model represents the state as position and momentum and introduces a trainable positive parameter α within a mechanical-dynamics formulation.Its stability analysis uses a total energy combining kinetic energy with the learned potential ε(q, w).Source: 2) Second–order model · pp. 2–3
• The second-order system eventually converges to a fixed point (q∗, 0_nv), where q∗ is a local minimizer of ε.A stability-preserving generalization is also defined using symmetric matrix conditions.Source: 2) Second–order model · p. 3

3) Stochastic Model:

• The stochastic model resembles a drift–diffusion process in which the energy ε(x, w) acts as a drift potential and β is an optimisable diffusion coefficient.Source: 3) Stochastic Model · p. 3
• The density follows a Kolmogorov forward equation and converges to a Boltzmann stationary distribution while the associated free-energy Lyapunov functional decays.Source: 3) Stochastic Model · p. 3
• Experimental evaluation of the stochastic model is left for future work because optimisation requires stochastic adjoint-sensitivity analysis.Source: 3) Stochastic Model · p. 3

III. TRAINING THE MODEL

• Training the model parameters is formulated by minimising a smooth data-fitting cost over the available input–output samples.Source: §III · p. 3

A. Training Process: an Optimal Control Perspective

• The training objective minimises a smooth cost over model parameters, with gradient descent providing a recursive approximation to a locally optimal solution.Source: A. Training Process: an Optimal… · p. 3
• Offline data use batch gradient descent, whereas sequential observations use stochastic gradient descent with the iteration counter tied to the data index.Source: A. Training Process: an Optimal… · p. 3
• With sufficiently small steps and enough iterations, gradient descent converges arbitrarily close to a local minimizer of the average training objective.Source: A. Training Process: an Optimal… · p. 3
• Adjoint sensitivity analysis computes parameter gradients with O(1) memory and supports both terminal costs and costs distributed across the depth domain.Source: A. Training Process: an Optimal… · p. 3

B. Terminal Cost Gradients

• For terminal costs, the adjoint system is solved backward and its initial value recovers the parameter gradient for stable first- and second-order neural ODE models.Source: B. Terminal Cost Gradients · pp. 3–4

C. Back–Propagated Cost Gradients

• Integral or back-propagated costs use an adjoint initial-value problem whose backward solution recovers the parameter gradient at the initial depth.Source: C. Back–Propagated Cost Gradients · p. 4
• The derivation follows the terminal-cost case by introducing the corresponding adjoint dynamics and recovering dℓ/dw from µ(0).Source: C. Back–Propagated Cost Gradients · p. 4

D. Gradients with Respect to Integration Bound

• The integration bound S can be learned alongside w using gradient-descent updates and represents the horizon of the optimal-control problem.Source: D. Gradients with Respect to… · p. 4
• For stable neural ODEs, the terminal-cost gradient with respect to the integration bound is directly correlated with the energy stored in the system at the end of integration.Source: D. Gradients with Respect to… · pp. 4–5

• Gradients for learned projection parameters are computed alongside the ODE parameters when input and output projections are optimized.Source: E. Gradients with respect to input and… · p. 5
• The derivations cover both terminal-cost and back-propagated-cost settings, with separate expressions for the corresponding projection gradients.Source: E. Gradients with respect to input and… · p. 5

• A terminal cost term is added to the loss because bounded-depth flows are not guaranteed to reach steady state.The regularizer is intended as a soft constraint that accelerates convergence.Source: F. On Training Regularisers · p. 5
• The same regularization term can also encourage regularity and stability in standard vanilla neural ODEs.Source: F. On Training Regularisers · p. 5

• All experiments use a Dormand–Prince adaptive-step solver with absolute and relative tolerances of 10^-6.Source: §IV · p. 5
• The scalar γ for the regularized loss is set to 10^-2 throughout the experiments.Source: §IV · p. 5

A. Function approximation

• The model learns the function y = −u by shaping a learned energy landscape whose minimum configuration represents the desired mapping.The energy function is conditioned on u_t and the learned flow follows steepest directions.Source: A. Function approximation · p. 5

B. Nonlinear classification via stable neural flows

• For classification, the ODE steers noisy class samples toward linearly separable clusters before a linear output projection computes the task cost.The setup includes half-moons and three-spirals datasets with distinct output losses.Source: B. Nonlinear classification via stable… · p. 5
• The half-moons variant uses a data-independent energy function, while the three-spirals variant uses a data-dependent stable neural flow.Source: B. Nonlinear classification via stable… · p. 5
• The classification flows correctly steer inputs into linearly separable clusters and converge to desired values before S := 1, confirming stability in these experiments.Source: B. Nonlinear classification via stable… · p. 5

V. CONCLUSION AND FUTURE WORK

• Stable neural flows replace an unconstrained vector field with an energy manifold whose gradient drives the data flows, with regularizers designed to accelerate steady-state convergence.Source: §V · p. 5
• The paper identifies future work involving observers and controllers based on neural ODEs.Source: §V · p. 6