Source-linked AI summary
On Second Order Behaviour in Augmented Neural ODEs
Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, Pietro Liò
TL;DR
Second-order laws govern many physical systems, but second-order behavior in Neural ODEs had not been studied in depth. The paper develops SONODE adjoint optimization and analyzes how SONODEs and ANODEs learn higher-order dynamics. It finds that their gradients can be computed equivalently through a more efficient coupled first-order ODE, and that SONODE inductive biases are generally beneficial on the studied systems.
Problem
Second-order laws govern many scientific systems, but second-order behavior in Neural ODEs had not received an in-depth study.
Source: §1 · p. 1Method
The paper extends adjoint sensitivity to SONODEs, analyzes ANODEs’ higher-order dynamics, and compares the models on real and synthetic systems.
Source: §1 · p. 2Results
Takeaways & Limitations
ANODEs can learn second-order dynamics, but entangled state and augmented dimensions constrain their interpretability in scientific applications.
Source: §5.3 · p. 6, §8 · p. 9ANODE dynamics entangle state and augmented dimensions, making the learned force equation less directly interpretable than SONODEs’ acceleration function.
Source: §5.3 · p. 6
Abstract
from arXiv · showhide
Neural Ordinary Differential Equations (NODEs) are a new class of models that transform data continuously through infinite-depth architectures. The continuous nature of NODEs has made them particularly suitable for learning the dynamics of complex physical systems. While previous work has mostly been focused on first order ODEs, the dynamics of many systems, especially in classical physics, are governed by second order laws. In this work, we consider Second Order Neural ODEs (SONODEs). We show how the adjoint sensitivity method can be extended to SONODEs and prove that the optimisation of a first order coupled ODE is equivalent and computationally more efficient. Furthermore, we extend the theoretical understanding of the broader class of Augmented NODEs (ANODEs) by showing they can also learn higher order dynamics with a minimal number of augmented dimensions, but at the cost of interpretability. This indicates that the advantages of ANODEs go beyond the extra space offered by the augmented dimensions, as originally thought. Finally, we compare SONODEs and ANODEs on synthetic and real dynamical systems and demonstrate that the inductive biases of the former generally result in faster training and better performance.
1 Introduction
The paper studies second-order dynamics in Neural ODEs, which are important for physical systems but have received little in-depth attention. It examines SONODEs and ANODEs, finding that SONODEs often learn simpler solutions and that their inductive biases are beneficial on the studied systems.
1 Introduction
- Second-order laws govern many scientific systems, yet second-order Neural ODE behavior had not received an in-depth study.Examples include Newton’s equations of motion and oscillators.Source: §1 · p. 1
- For the compact parity experiment, NODEs fail to learn the mapping, ANODE(1) succeeds, and SONODEs learn the simplest trajectory.The experiment is identified as g1d from the original work [4].Source: Figure 1 · p. 2
- The authors study low-dimensional physical systems, often with known analytic solutions, to analyze model properties.This differs from prior work focused mainly on classification tasks.Source: §1 · p. 2
- SONODEs and ANODEs are compared on real and synthetic second-order systems, where SONODE inductive biases are generally beneficial.The paper also reports that ANODEs can learn higher-order dynamics with few augmented dimensions, but that their entangled dynamics affect interpretability and the loss landscape.Source: §1 · p. 2
2 Background
The background frames NODEs as continuous-depth models derived from the ResNet perspective and trained with black-box ODE solvers. The paper focuses on learning dynamical-system trajectories from sampled time points, primarily using mean squared error.
2 Background
- Neural ODEs are a continuous variant of ResNet models, with hidden states evolving continuously according to a differential equation; gradients can be computed through an abstract adjoint state once its dynamics are known [3].Source: §2 · p. 2
- ANODEs append augmented states, and this paper allows their initial values to be learned from the initial state through a neural network.ANODE(D) denotes the use of D augmented dimensions.Source: §2 · p. 2
- For trajectory modeling, training uses mean squared error over predicted locations at sampled times; toy classification instead uses cross-entropy on final-position separability.The trajectory data comprise N + 1 sample points from fixed trajectories at multiple time steps.Source: §2 · p. 2
3 Second Order Neural Ordinary Differential Equations
SONODEs model acceleration with a neural network and can be expressed as a constrained coupled first-order ODE. The paper extends adjoint sensitivity to this setting and shows that coupled first-order optimization gives equivalent parameter gradients with fewer matrix multiplications.
3 Second Order Neural Ordinary Differential Equations
- A SONODE specifies initial position and velocity, with acceleration given by a neural network of position, velocity, and time.It can also be written as a coupled first-order neural ODE with state z(t) = [x(t), a(t)].Source: §3 · p. 3
- SONODEs are ANODEs with structural constraints on the vector field, allowing reuse of NODE’s first-order adjoint method while motivating a second-order adjoint derivation.The paper generalizes adjoint sensitivity to SONODEs and compares the resulting gradients with those of the coupled first-order ODE.Source: §3 · p. 3
- The second-order adjoint follows a second-order ODE, and the paper provides an alternative proof of the first-order adjoint result.The adjoint state over time supports training the vector-field parameters.Source: §3 · p. 3
- The coupled first-order ODE and second-order adjoint yield equivalent gradients for θf, while the second-order adjoint requires at least as many matrix multiplications.The paper therefore uses the first-order coupled formulation for its experiments; entangled adjoint representations permit its faster gradient computation.Source: §3 · p. 3
4 Properties of SONODEs
The section establishes phase-space constraints for SONODE trajectories and tests how SONODEs, NODEs, and ANODEs handle parity and nested-class problems. Results show SONODEs can cross in real space and achieve the lowest parity loss, while ANODEs and SONODEs can learn the nested-spheres mapping.
4.1 Generalised parity problem
- For a k-th order IVP with Lipschitz-continuous highest derivative and no explicit time dependence, unique phase-space trajectories cannot intersect at an angle.A single phase-space trajectory also cannot intersect itself at an angle.Source: §4.1 · p. 4
- SONODEs can learn a parity flip in any dimension through straight-line trajectories from each point through the origin to its negation.The stated trivial solution uses zero acceleration and a chosen initial velocity.Source: §4.1 · p. 4
- NODE parity flips work by pairing dimensions for rotations in even dimensions, while odd dimensions leave one dimension unrotated; increasing dimensionality is expected to ease the task.The expected easing is attributed to exponentially decreasing point density when the number of data points stays constant.Source: §4.1 · p. 4
- SONODEs have the lowest loss across the tested parity generalisations; NODE loss oscillates across dimensions, while ANODEs perform better especially in odd dimensions.The experiment uses 50 training and 10 test points, and the SONODE result is associated with a trivial solution available in any dimension.Source: §4.1 · p. 4, Figure 2 · p. 4
4.1.1 Nested n-spheres
- In the nested-spheres trajectories, NODEs preserve topology, ANODEs use a third dimension to separate the regions, and SONODE points pass through one another in real space.The comparison is shown for the problem in 2D.Source: Figure 3 · p. 4
- SONODEs are not restricted to homeomorphic transformations in real space, and the nested-spheres experiment finds that only ANODEs and SONODEs learn the mapping.The task surrounds the blue class with the red class, preventing linear separation by a homeomorphic transformation in that space.Source: §4.1.1 · pp. 4–5
5 Second order behaviour in SONODEs and ANODEs
ANODEs can represent higher-order dynamics through learned, entangled functions, sometimes with fewer augmented dimensions than the state dimension. Compared with ANODEs, SONODEs directly learn acceleration and initial velocity, yielding more interpretable dynamics and a more constrained functional solution.
5.1 How do ANODEs learn second order dynamics?
- ANODEs learn an abstract function that equals the initial velocity at t0 and a coupling function that supplies the right acceleration, whereas SONODEs learn acceleration and initial velocity directly.ANODE’s state and augmented dimension are entangled, so its augmented dimension is not itself the velocity.Source: §5.1 · pp. 5–6
5.2 Minimal augmentation
- With fewer augmented than state dimensions, ANODE can solve an overdetermined system when a solution exists; otherwise, its pseudo-inverse gives the best least-squares approximation.The paper notes that the approximation improves as the number of augmented dimensions approaches the state dimension.Source: §5.2 · p. 6
- ANODE(1) learns a 2D second-order function with one augmented dimension, whose trajectory differs greatly from the ODE’s velocity.This demonstrates that ANODE need not double the state dimensions to learn this function.Source: §5.2 · p. 6, Figure 5 · p. 6
5.3 Interpretability of ANODEs
- Across two initialisations, both models learn the real-space trajectory, but only SONODE recovers the true velocity; ANODE’s augmented trajectories differ widely from it.The comparison concerns a 2D ODE.Source: §5.3 · p. 7, Figure 6 · p. 7
5.4 The functional loss landscape
- ANODEs have infinitely many non-trivial functional forms that model the true second-order dynamics, while SONODE’s constraints yield a unique functional form.The preceding experiment reports ANODE converging to different augmented trajectories across random initialisations, while SONODE converges to the underlying ODE.Source: §5.4 · p. 7
6 Experiments on second order dynamics
The experiments compare SONODEs and ANODEs on synthetic and real dynamical systems. Results vary by setting: SONODEs learn harmonic dynamics faster and extrapolate better on noisy sine curves and Silverbox, while ANODEs perform marginally better on airplane vibrations.
6.1 Synthetic harmonic oscillators and noise robustness
- On damped harmonic oscillators, SONODEs converge faster because second-order behavior is built in, while all models reduce loss to approximately zero.The experiment used 30 random initial position-and-velocity pairs.Source: §6.1 · pp. 7–8, Figure 7 · p. 7
- On noisy sine-curve extrapolation, SONODEs are more robust to noise and extrapolate better than ANODEs.Models trained on 50 noisy points from the first 10 seconds and were tested on 10 points in the next 5 seconds, with training noise standard deviations from 0 to 0.7.Source: §6.1 · p. 8, Figure 8 · p. 8
6.2 Experiments on real-world dynamical systems
- On real systems, SONODEs extrapolate better on Silverbox, whereas ANODEs perform marginally better on higher-order airplane vibrations.The Silverbox models trained on the first 1,000 timestamps and extrapolated over the next 4,000; the airplane-vibration models used the same split.Source: §6.2 · p. 8, Figure 9 · p. 8, Figure 10 · p. 9
7 Discussion and related work
SONODEs restrict ANODE phase-space dynamics to second-order behavior, favoring time-series tasks with known second-order dynamics. The discussion contrasts this setting with tasks driven by endpoints and situates the work among other second-order and physics-based models.
SONODEs vs ANODEs
- For time-series data assumed to have second-order dynamics, the authors expect SONODEs to improve accuracy and convergence, with fewer local minima and greater parameter efficiency.For endpoint-based tasks such as classification, they suggest unconstrained ANODEs might perform better.Source: §7 · p. 9
Second Order Models
Physics Based Models
- Related physics-based approaches use Lagrangian or Hamiltonian mechanics to learn dynamical data, in discrete and continuous settings.The passage cites Lutter et al., Greydanus et al., and Zhong et al..Source: §7 · p. 9
8 Conclusion
The paper extends adjoint sensitivity methods to SONODEs and studies how SONODE and ANODE dynamics behave in modelling tasks. It concludes that SONODE inductive biases generally benefit second-order modelling, while ANODEs can learn such dynamics with interpretability costs.
8 Conclusion
- The paper generalizes the adjoint sensitivity method to SONODEs and compares it with training the equivalent coupled ODE.It also studies SONODE theoretical properties through toy modelling problems.Source: §8 · p. 9
- ANODEs can learn second-order dynamics, but do so by approximating an abstract coupled ODE whose state and augmented dimensions become entangled in velocity.The paper links this mechanism to interpretability concerns in scientific applications and to the shape of the loss landscape.Source: §8 · p. 9
- Across the studied synthetic and real second-order systems, SONODE inductive biases are generally beneficial; the underlying principles can extend to higher orders.The authors state that higher-order extensions retain faster training and better modelling performance, with a proof of principle in Appendix E.2.Source: §8 · p. 9
Broader Impact
The broader impact discussion points to possible uses of SONODEs in physical systems and time-series applications. It characterizes the work's contribution as incremental while Neural ODEs' potential remains uncertain.
Broader Impact
- The work's theoretical results are expected to matter through SONODE applications, including physical systems described by Newton's second law.The authors mention future time-series applications such as medical-record evolution, earthquake monitoring, and traffic forecasting.Source: Broader Impact · p. 10
- The authors identify possible application areas in oscillator circuits, mechanical systems such as car suspension, and airplane-vibration modelling.They describe possible uses including circuit and signal monitoring, car safety, and aircraft design.Source: Broader Impact · p. 10
- The authors characterize the contribution as incremental because Neural ODEs' full potential and future applications remain uncertain.They note that future applications could have both positive and negative uses.Source: Broader Impact · p. 10
A Phase Space Trajectory Proofs
The appendix establishes phase-space trajectory constraints and derives adjoint sensitivity methods for first- and second-order dynamics. It also shows how the second-order adjoint can be consolidated into a single adjoint state.
A Phase Space Trajectory Proofs
- For a k-th order IVP with a Lipschitz-continuous, time-independent highest derivative, unique phase-space trajectories cannot intersect each other or themselves at an angle.The proof uses continuity of the trajectories and equality of their derivatives at a shared phase-space point.Source: A Phase Space Trajectory Proofs · p. 12
B Adjoint Sensitivity Method
- A Lagrangian-style derivation proves both first- and second-order adjoint methods and shows that the second-order method’s adjoint can be expressed as a single state.The first-order adjoint derivation is presented as extensible to second-order ODEs.Source: B Adjoint Sensitivity Method · p. 13, §B.1 · p. 14, §B.2 · p. 18
B.1 First Order Adjoint Method
- The first-order adjoint method computes gradients by integrating the state and adjoint backward in time, without storing intermediate values and with constant memory.The derivation generalizes an earlier first-order adjoint derivation to provide a route to the second-order case.Source: §B.1 · p. 14
B.2 Second Order Adjoint
- The second-order adjoint state of a SONODE follows a second-order ODE, derived by adding the ODE and initial-condition constraints before differentiating and integrating by parts.Position- and velocity-dependent loss gradients can be combined into a new adjoint state.Source: §B.2 · pp. 15, 17, 18
B.3 Equivalence between the two Adjoint methods
For second-order dynamics, the first-order adjoint on the concatenated state yields the same gradients with no more matrix multiplications. The appendix also shows that second-order ODE transformations in real space need not be homeomorphisms.
B.3 Equivalence between the two Adjoint methods
- The first-order adjoint on z = [x, v] produces the same gradients as the second-order adjoint, while requiring at least as many matrix multiplications for the latter.The first-order method requires equal or fewer matrix multiplications for gradients at each step, initial conditions, and final gradients.Source: §B.3 · pp. 18, 21, 22
- The coupled first-order adjoint is more efficient because its representation entangles adjoint information, whereas the second-order representation separates the adjoint state and its velocity.The paper states that the entangled representation computes gradients more efficiently because it need not be interpretable.Source: §B.3 · p. 22
C Second Order ODEs are not Homeomorphisms
- A second-order ODE transformation in real space is not always one-to-one and therefore is not always a homeomorphism.Source: C Second Order ODEs are not… · p. 22
D.1 Functional Form Proofs
The appendix contrasts ANODEs’ many function forms for representing second-order dynamics with SONODEs’ unique form. It also shows that ANODEs can learn multiple functions under a shared ODE by changing their initial conditions.
D.1 Functional Form Proofs
- The stated ANODE solution for G requires explicit dependence of F on augmented dimensions, with ∂F/∂a^T having a left inverse.Without explicit a-dependence, the required inverse does not exist.Source: §D.1 · p. 23
- ANODEs can learn infinitely many non-trivial functional forms for the same second-order dynamics, whereas SONODEs have a unique form.The ANODE result follows from a gauge symmetry; the SONODE result concerns approximating the true dynamics in real space.Source: §D.1 · pp. 23–24
- ANODE dynamics are preserved when an arbitrary φ(x) with φ(x0) = 0 is added to F and G is changed correspondingly.The resulting system retains the initial velocity.Source: §D.1 · pp. 23–24
- The proof considers losses depending only on position and notes that losses depending on both position and velocity would impose more restrictions.The text argues that validity for position-only loss also establishes validity when loss depends on both.Source: §D.1 · p. 24
- The SONODE uniqueness proof uses matching position trajectories to infer matching velocities and accelerations, forcing the same dynamics and initial velocity.The argument assumes the compared systems share their position over time.Source: §D.1 · pp. 23–25
D.2 ANODEs Learning Two Functions
- ANODEs learn two functions with mixed sine and cosine components using one shared ODE and different initial conditions.The text also reports success when a third function is added.Source: §D.2 · p. 25
- For the two-function construction, the gauge change preserves the ODE while changing the initial-velocity expression; two equations determine its two unknowns.The same approach was extended to a third function.Source: §D.2 · p. 25
E Experimental Setup and Additional Results
Across these additional experiments, SONODEs use fewer function evaluations than NODEs at the same MNIST accuracy, while ANODEs fit the forced Van der Pol training data better but extrapolate worse. TONODEs underperform on Airplane Vibrations, and all three model types learn the tested first-order exponential.
E Experimental Setup and Additional Results
- The experiments use full datasets without minibatches and repeat runs three times; physical-system tasks use Adam at learning rate 0.01, and model architectures vary by task.For system-identification tasks, the setup uses a single linear layer; other tasks use a two-hidden-layer network.Source: E Experimental Setup and Additional… · pp. 25–26
E.1 Van Der Pol Oscillator
- On the forced Van der Pol oscillator, ANODEs achieve lower training loss but greater test loss than SONODEs; neither learns the chaotic system particularly well.The authors conjecture that ANODEs overfit and attribute the difficulty for both models to the system’s chaotic behaviour.Source: Figure 12 · p. 26, §E.1 · p. 27
E.2 Third Order NODEs on Airplane Vibrations
- TONODEs underperform ANODEs and SONODEs on Airplane Vibrations, with large error on both training data and extrapolation.Across three repetitions, initialization led to a zero solution; the authors hypothesize that rescaling time to [0, 1] could improve accuracy.Source: §E.2 · p. 27, Figure 13 · p. 27
E.3 First Order Dynamics and Interpolation
- NODEs, SONODEs, and ANODEs all learn the tested noise-free exponential, consistent with SONODEs containing NODEs as a subset and ANODEs encompassing SONODEs.The experiment tests x(t) = exp(0.1667t); all models learn the function.Source: §E.3 · p. 27, Figure 14 · p. 28
E.4 Performance on MNIST
- SONODEs reach higher MNIST test accuracy than NODEs with lower NFE, while ANODEs reach the same higher accuracy with higher NFE and fewer parameters than SONODEs.The table reports results at convergence; Figure 15 also says SONODEs use lower NFE than NODEs.Source: Table 1 · p. 28, §E.4 · p. 28, Figure 15 · p. 28