PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Flows: Efficient Alternative to Neural ODEs

Marin Biloš, Johanna Sommer, Syama Sundar Rangapuram, Tim Januschowski, Stephan Günnemann

arXiv:2110.13040v1cs.LGcs.NAmath.NA

TL;DR

Neural ODEs model evolving systems but require numerical solving, which can be computationally expensive. This paper models solution curves directly with neural networks and reports better performance than ODE-based models in time series modeling and density estimation, at lower computation cost.

  • Problem

    Neural ODEs model continuous-time dynamics, but their numerical solutions can be costly, and learning an unknown vector field with a neural network makes the ODE non-interpretable.

    Source: §1 · p. 1

  • Method

    Neural flows parameterize ODE solution curves directly with a neural network, subject to conditions that make the function a valid flow.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Neural flows outperform ODE-based models in time series modeling and density estimation at much smaller computation cost.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The reduced computation cost may make it possible to scale to larger datasets and models in the future.

    Source: §5 · p. 9

    For autonomous ODEs with a fixed time-independent vector field, the paper cannot define a general neural flow satisfying that condition.

    Source: §5 · p. 9

Abstract

from arXiv · show

Neural ordinary differential equations describe how values change in time. This is the reason why they gained importance in modeling sequential data, especially when the observations are made at irregular intervals. In this paper we propose an alternative by directly modeling the solution curves - the flow of an ODE - with a neural network. This immediately eliminates the need for expensive numerical solvers while still maintaining the modeling capability of neural ODEs. We propose several flow architectures suitable for different applications by establishing precise conditions on when a function defines a valid flow. Apart from computational efficiency, we also provide empirical evidence of favorable generalization performance via applications in time series modeling, forecasting, and density estimation.

1 Introduction

Neural ODEs model time-varying systems with a learned vector field, but solving them numerically can be costly. Neural flows instead model solution curves directly, returning solutions in one network pass without ODE solvers.

1 Introduction

  • Numerical ODE solving trades computation cost against precision, motivating work to reduce solver evaluations while retaining expressive dynamics.A continuous vector field needs only continuity in time and Lipschitz continuity in the state for a unique solution [14].Source: §1 · p. 1
  • In machine learning, the vector field is unknown and learned from data, but representing it with a neural network makes the ODE non-interpretable.One example is evolving a latent state in continuous time to predict measurements in a time series [16].Source: §1 · p. 1
  • Neural flows model ODE solution curves directly with a neural network, returning solutions in one forward pass without numerical solvers.The paper proposes this as a faster alternative to ODE-based models while retaining their modeling capability.Source: §1 · p. 2

2 Neural flows

Neural flows model ODE solution curves directly, with conditions that make the modeled function a valid flow. The paper presents architectures for continuous-time dynamics and discusses their approximation capabilities and practical flexibility.

2 Neural flows

  • A neural flow F(t, x0) models an ODE solution curve and must preserve the initial condition and be invertible at each time.These properties ensure an exact correspondence between F and an ODE vector field; arbitrary-time initial conditions can be handled by recovering x(0) with F^-1(t0, x0).Source: §2 · p. 2

2.1 Proposed flow architectures

  • GRU flow defines continuous-time hidden-state evolution and is a valid flow when its component networks are contractive and α = 2.The paper states that this construction makes the network invertible and retains GRU-ODE's bounded hidden state and Lipschitz constant of 2.Source: §2.1 · p. 3
  • Coupling flow transforms one partition of the input conditioned on the copied other partition, giving an invertible transformation by design.Multiple layers use different partitions because some values stay constant in a single layer.Source: §2.1 · p. 3
  • The architectures can be composed while preserving flow properties, and their time embeddings are chosen to be zero at t = 0.The paper uses linear or tanh time dependence in experiments and also describes Fourier features.Source: §2.1 · p. 3

2.2 On approximation capabilities

  • ResNet flow can approximate ODE solutions through Euler discretization, while finitely many layers limit the practical value of the approximation result.The paper also notes that flow conditions can be relaxed for a task, such as removing a Lipschitz constraint when invertibility is unnecessary.Source: §2.1 · p. 2, §2.2 · pp. 3–4

3 Applications

Section 3 adapts neural flows to irregular-time latent-variable models, temporal point processes, and time-dependent density estimation. Across these applications, flow evaluation replaces ODE solves in selected model components while supporting continuous-time predictions and densities.

3.1 Continuous-time latent variable models

  • The smoothing model infers q(z0|X, t) variationally and decodes latent states deterministically, while filtering updates p(zi|x1:i, t1:i) online.Filtering uses the posterior at the final observed time to evolve predictions, rather than evolving from the initial posterior.Source: §3.1 · pp. 4–5
  • Neural flows replace ODE solving with one network pass per observation in continuous-time latent models, supporting irregular measurements and predictions at arbitrary times.The paper notes that this computational efficiency matters as data scale grows.Source: §3.1 · pp. 4–5

3.2 Temporal point processes

  • For temporal point processes, the model represents event history and predicts a positive conditional intensity; marked processes additionally predict event-type probabilities from history.The marked-process formulation models arrival times and event marks jointly.Source: §3.2 · p. 5

3.3 Time-dependent density estimation

  • In high-dimensional data, continuous normalizing flows require stochastic trace estimation, in addition to solver computation.The paper notes that trace and flow determinant calculations are equivalent for linear ODEs.Source: §3.3 · p. 6
  • For spatio-temporal data, affine coupling flow models p(xi|ti) at varying event times using the inverse flow and Jacobian determinant, and generates samples by evaluating F(ti, x0).This extends the coupling-flow density model to evolving event-location densities without solver or trace estimation.Source: §3.3 · p. 6
  • An attention encoder conditions each event density on prior values; ResNet-flow variants lose their ODE advantage because inversion and determinant calculation require iteration.Source: §3.3 · p. 6

4 Experiments

Across time-series, forecasting, temporal point-process, and spatial-density experiments, neural flows match or outperform ODE-based models at lower computational cost. They also handle the tested stiff dynamics where an adaptive ODE solver misses the solution.

Synthetic data

  • On synthetic data, neural flows are an order of magnitude faster with fixed steps and perform better on non-smooth signals and extrapolation, though neither model excels at extrapolation.Neural flows also output only smooth dynamics, and fixed-step ODE solvers do not guarantee invertibility.Source: §4 · p. 6

Smoothing approach

  • Neural flows outperform ODE models across reconstruction and prediction tasks, with Physionet reconstruction within the confidence interval.The models use approximately the same number of trainable parameters.Source: Table 1 · p. 7, §4 · p. 7

Stiff ODEs

  • For the tested stiff ODE, the adaptive-solver neural ODE misses the correct solution, whereas the flow captures it.The flow evaluates the solution at the initial time and then at the target time without a numerical solver.Source: §4 · p. 7, Figure 2 · p. 7

Speed improvements

  • In smoothing experiments, neural flows provide more than a twofold training speed-up over an Euler-method ODE, with a larger gap for adaptive solvers.The reported epoch time includes operations beyond ODE solving, such as data fetching and state updates.Source: §4 · p. 7

Filtering approach

  • The GRU flow mostly outperforms GRU-ODE on healthcare forecasting and requires 60% less time per training epoch.The ordinary ResNet flow performs worse; the authors suggest it may lack GRU flow properties such as boundedness.Source: §4 · p. 7, Table 2 · p. 8

Temporal point processes

  • On real-world TPP data, neural flows generally outperform ODE models except for one ODE model on Wiki; mixture distributions usually beat corresponding continuous-intensity models.Continuous encoders are preferred to discrete RNNs, while continuous intensity offers no benefit and is much slower; flow models are over an order of magnitude faster than ODEs with continuous intensity.Source: Table 3 · p. 8, §4 · p. 8

Spatial data

  • Continuous coupling normalizing flows achieve better test NLL on all spatial datasets and remain more than an order of magnitude faster.Because affine coupling is simple, these models require more parameters.Source: §4 · p. 8, Table 4 · p. 9

5 Discussion

Neural flows model ODE solution curves directly, avoiding numerical solvers while retaining useful modeling properties. The discussion reports favorable efficiency and performance, while noting limits for exact ODE recovery and autonomous flows.

5 Discussion

  • Neural flows outperform ODE-based models in time-series modeling and density estimation at much smaller computation cost.The authors suggest this may allow scaling to larger datasets and models.Source: §5 · p. 9
  • Unlike approaches that approximate solutions to a specified ODE, neural flows learn solutions corresponding to the unknown ODE and guarantee a proper flow.The guarantee is required for certain applications.Source: §5 · p. 9
  • Exact flows cannot always be found for ODEs without closed-form solutions, and the proposed architectures do not generally guarantee flows for autonomous ODEs.The paper discusses a regularization-based potential solution for the autonomous case.Source: §5 · p. 9
  • Neural ODEs can be more parameter-efficient because they reuse the same function in the solver, although flows remain more efficient in the reported density-estimation setting.The authors note that representing the same dynamics may sometimes require more flow parameters.Source: §5 · p. 9
  • The discussion identifies other flow architectures, additional use cases, and higher-order dynamics as directions for future work.The paper has explored time-series modeling and density estimation, and the authors mention adapting flows to other applications.Source: §5 · p. 9
  • The authors flag privacy and fairness as considerations for sensitive applications such as healthcare, and note reduced computation cost may imply energy savings.The paper used several healthcare datasets.Source: §5 · p. 9

A.1 GRU-ODE definition

  • GRU flow is presented as retaining GRU-ODE's bounded hidden state and Lipschitz continuity without numerical solvers.The hidden state stays in (−1, 1), and the transformation has Lipschitz constant at most 2.Source: §A.1 · p. 14

A.2 Training loss for GRU-ODE-Bayes

The appendix describes the GRU-ODE-Bayes loss, GRU-flow properties, ODE reparameterization, attentive normalizing flows, autonomous-flow constraints, and computation costs for continuous normalizing flows.

A.2 Training loss for GRU-ODE-Bayes

  • The GRU-ODE-Bayes objective combines masked negative log-likelihood with a Bayesian KL-divergence term that is closed-form for normal distributions.The Bayesian update combines the predicted distribution with the observation-noise distribution.Source: §A.2 · p. 14

A.3 Proof of Theorem 1

  • The GRU flow is invertible when its added second summand is contractive, with a Lipschitz constant below one.The proof uses bounded time dependence ϕ(t) ∈ [0, 1] that does not depend on the hidden state.Source: §A.3 · p. 15

A.3.1 Properties of GRU flow

  • GRU flow keeps the hidden state in (−1, 1) and gives the full transformation a Lipschitz constant no greater than 2.These are the stated boundedness and continuity properties.Source: §A.3.1 · p. 15

A.4 ODE reparameterization

  • ODE reparameterization lets a batch be solved in parallel, and the authors use it in all experiments.The cited approach maps each example's time interval to s ∈ [0, 1].Source: §A.4 · p. 16

A.5 Attentive normalizing flow

  • The attentive normalizing flow conditions each element on preceding elements, using X1:i−1 rather than the X1:i context used by the cited attentive CNF.Diagonal masking preserves dependence on past values rather than future ones.Source: §A.5 · p. 16

A.6 Autonomous ODEs

  • The proposed architectures do not generally satisfy the extra condition for autonomous flows, and the paper reports no general neural-flow parameterization that captures all autonomous ODEs.A regularization penalty is offered as a way to learn the desired autonomous behavior rather than guarantee it.Source: §A.6 · p. 17

A.7 Linear ODE and change of variables

  • For a linear ODE, the matrix-exponential flow yields the transformed density by multiplying q(z) by |det exp(A)|^-1, equivalently exp(tr(A))^-1.The passage presents this as the discrete change-of-variables result for the linear system.Source: §A.7 · p. 17

A.8 Computation complexity of (continuous) normalizing flows

  • Exact trace evaluation in continuous normalizing flows costs O(d^2) per solver step, while the cited approximation costs O(d) but makes training noisier.Exact Jacobian determinants generally cost O(d^3), whereas triangular Jacobians allow O(d) determinant computation.Source: §A.8 · p. 18

B Synthetic experiments

The synthetic experiments evaluate neural ODEs and flows on periodic signals and data generated from known autonomous ODEs, with extrapolation tests described for the datasets.

B Synthetic experiments

  • The synthetic test set includes periodic signals, including a sine system with flow F(t, x) = x + sin(t), and data generated from known ODEs.The known-ODE datasets include a Lotka–Volterra parametrization.Source: B Synthetic experiments · p. 18
  • For the periodic-signal setup, initial values are sampled from (−2, 2) over times (0, 10), with extrapolation extending the initial-value range to (−4, 4) and time to 30.For the Lotka–Volterra data, the stated extrapolation initial conditions are x1, x2 ∈ [1, 2].Source: B Synthetic experiments · p. 18
  • Figure 7 compares fixed and adaptive solvers on synthetic data: fixed solvers train faster while retaining similar accuracy.The caption assigns training time to the left panel and accuracy to the right.Source: Figure 7 · p. 19

B.1 Comparing adaptive and fixed-step solvers

On synthetic data, fixed-step solvers train faster than adaptive solvers, but the density-estimation example shows that a better fixed-solver score can come from an invalid density.

B.1 Comparing adaptive and fixed-step solvers

  • Adaptive solvers require significantly longer training times than fixed-step solvers across the tested model configurations.The reported adaptive-solver run used default tolerances rtol = 10^-7 and atol = 10^-9, illustrating the speed–numerical-accuracy trade-off.Source: §B.1 · p. 19
  • Fixed solvers can achieve similar or better results with less computation, but may produce overlapping trajectories and non-unique solutions.The passage identifies this as a potential problem for applications such as density estimation.Source: §B.1 · p. 19
  • Euler's estimated area under the density curve is 1.06, so it does not define a proper density despite achieving a better score.The authors say the extra mass can produce better results without capturing the true distribution better.Source: Figure 8 · p. 19, §B.1 · p. 19

B.2 Comparing flow configurations

The section compares neural-flow configurations and reports their synthetic-data fit alongside comparisons with neural ODEs in forecasting, extrapolation, and training time.

B.2 Comparing flow configurations

  • The comparison tested coupling and ResNet models with linear or tanh ϕ, with and without weight regularization, across two or four transformations.Each transformation used a two-hidden-layer network with 64 hidden dimensions; configurations were run five times.Source: §B.2 · p. 19
  • All tested models captured the synthetic data perfectly except the coupling flow with linear time function ϕ, which did not converge.The authors suggest large time inputs as a possible cause and say different initialization or time normalization can fix the issue.Source: §B.2 · p. 20
  • Neural flows outperform neural ODEs in forecasting and extrapolation, and are faster during training.Tables 5–7 cover synthetic-data error, extrapolation from initial conditions outside the training distribution, and last-epoch wall-clock time, respectively.Source: §B.2 · p. 20, Table 5 · p. 20, Table 6 · p. 20

C Additional results

The additional results report training-time comparisons and full negative log-likelihood results for temporal point-process models, including marked models.

C Additional results

D.1 Encoder-decoder datasets

The section describes datasets and preprocessing for encoder-decoder experiments and MIMIC clinical data, including the construction of a filtered ICU cohort.

D.1 Encoder-decoder datasets

  • The MuJoCo dataset contains 10,000 sequences generated by sampling initial body positions, limb positions, and velocities; the authors use it unchanged.Initial body positions are sampled in R2 from [0, 0.5], limbs from [−2, 2], and velocities from [−5, 5].Source: §D.1 · p. 21
  • The activity dataset groups similar positions into seven classes, while its time measurements are rounded to 100 ms intervals.The rounding reduces the union of points when batching, but is unnecessary for the flow models and for ODE reparameterization.Source: §D.1 · p. 21
  • The PhysioNet Challenge 2012 task is ICU mortality prediction; features are normalized to [0, 1], and timestamps are rounded to one minute.The data split is 60%–20%–20%; reconstruction MSE is scaled by 10^3 for this dataset and others, and by 10^2 for activity.Source: §D.1 · p. 22

D.2 MIMIC-III and MIMIC-IV

  • MIMIC-IV 1.0 contains de-identified ICU records from 2008–2019; after patient selection, the study retains 17,874 patients.MIMIC-III processing follows [16], and MIMIC-IV is processed similarly.Source: §D.2 · p. 22
  • The MIMIC data are organized into chart-events, inputs, outputs, and prescriptions; medications are sampled in 30-minute windows and longer administrations are split into fixed-time injections.For other tables, the authors keep only the most commonly used entries.Source: §D.2 · p. 22

D.3 TPP datasets

The temporal point-process experiments use synthetic and real-world event sequences, with inter-event times normalized and log-transformed for numerical stability.

D.3 TPP datasets

  • Four synthetic datasets each contain 1,000 sequences of 100 events: Poisson, log-normal renewal, and two Hawkes configurations.The Hawkes configurations differ in their stated baseline, component count, α, and β parameters.Source: §D.3 · pp. 22–23
  • The Reddit dataset contains 1,000 truncated sequences, with 984 subreddits used as marks.Sequences are capped at 100 events to make training with ODE-based models feasible.Source: §D.3 · p. 23
  • MOOC contains 7,047 sequences of up to 200 events, with 97 interaction types as marks.Source: §D.3 · p. 23
  • Wiki contains 1,000 page sequences of up to 250 edits, with 984 users used as marks.Source: §D.3 · p. 23
  • For real-world data, inter-event times are divided by the training-set empirical mean and then log-transformed as log(τ + 1).The log transform is included in the negative log-likelihood through the change-of-variables formula to improve numerical stability during training.Source: §D.3 · p. 23

D.4 Spatial datasets

The appendix specifies spatial datasets and preprocessing for density estimation, followed by optimizer and model configurations for smoothing, filtering, and density estimation experiments.

D.4 Spatial datasets

  • The spatial time-dependent density experiment reuses Chen et al.'s datasets and preprocessing pipeline [9].Further dataset details are referred to Chen et al. [9].Source: §D.4 · p. 23
  • The spatial datasets comprise Japanese earthquakes, New Jersey county COVID-19 cases, and New York bike rentals, each organized into sequences and train, validation, and test splits.Earthquake sequences span 30 days with 7-day gaps; COVID-19 sequences span 7 days; bike sequences cover one day starting at 5am. Spatial values are normalized to zero mean and unit variance, and time to [0, 1].Source: §D.4 · p. 23

E Hyperparameters

Smoothing experiments

  • Smoothing experiments use batch size 100, learning rate 1e-3 with 0.5 decay every 20 epochs, and three hidden layers; flow models use ResNet or coupling flows with one or two layers.The flow time function is tanh for ResNet and linear for coupling; dataset-specific encoder-decoder, latent-state, and GRU dimensions are listed for MuJoCo, Activity, and Physionet.Source: Smoothing experiments · pp. 23–24

Filtering experiment

  • Filtering experiments use batch size 100, learning rate 1e-3 with 0.33 decay every 20 epochs, and hidden dimension 64 on MIMIC-III or MIMIC-IV.ODE models use Euler or dopri5 solvers; flow models are GRU flows or ResNet flows.Source: Filtering experiment · p. 24
  • The filtering setup also includes marked and unmarked TPP experiments, with continuous or mixture ODE and flow models and GRU RNN baselines.Datasets are Reddit, MOOC, or Wiki; configurations include batch size 50, learning rate 1e-3, and hidden dimension 64.Source: Filtering experiment · p. 24

Density estimation experiment

  • Density estimation compares time-varying and attentive CNFs and NFs using batch size 50, learning rate 1e-3, and hidden dimension 64.CNFs use four hidden layers; coupling normalizing flows use four or eight base-density and time-dependent layers.Source: Density estimation experiment · p. 24
Loading 2110.13040v1…