Liquid Time-constant Networks

Ramin Hasani; Mathias Lechner; Alexander Amini; Daniela Rus; Radu Grosu

Source-linked AI summary

Liquid Time-constant Networks

Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, Radu Grosu

arXiv:2006.04439v4Jun 8, 2020cs.LGcs.NEstat.ML

TL;DR

Time-continuous recurrent models lacked a clear approach for combining adaptive dynamics with expressive, stable behavior. The paper introduces Liquid Time-Constant Networks, which use gated linear ODEs with input-dependent time constants, and reports stronger time-series performance than other RNNs across diverse tasks.

Problem
Time-continuous neural networks need rigorous measures for comparing expressive power, alongside evidence of stable dynamics and time-series approximation capability.
Method
LTCs combine linear first-order ODEs with nonlinear interlinked gates that produce input-dependent time constants, analyzed through theoretical bounds, trajectory length, and time-series experiments.
Results
LTCs improved performance by 5% to 70% over other RNN models in four of seven experiments and performed comparably in the remaining three.
Takeaways & Limitations
LTCs provide bounded and stable dynamics, greater measured expressivity, and improved supervised time-series prediction performance within the evaluated scope.
Takeaways & Limitations
Like many time-continuous models, LTCs exhibit vanishing gradients when trained by gradient descent, limiting their handling of long-term dependencies.

Abstract

from arXiv · show

We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems modulated via nonlinear interlinked gates. The resulting models represent dynamical systems with varying (i.e., liquid) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations, and give rise to improved performance on time-series prediction tasks. To demonstrate these properties, we first take a theoretical approach to find bounds over their dynamics and compute their expressive power by the trajectory length measure in latent trajectory space. We then conduct a series of time-series prediction experiments to manifest the approximation capability of Liquid Time-Constant Networks (LTCs) compared to classical and modern RNNs. Code and data are available at https://github.com/raminmh/liquid_time_constant_networks

1 Introduction · 2 LTCs forward-pass by a fused ODE solvers

LTCs define continuous-time recurrent dynamics using linear first-order ODEs modulated by nonlinear, input-dependent gates, producing liquid time-constants and bounded, expressive behavior. Their forward pass uses a fused explicit–implicit Euler solver designed for stiff LTC equations, with computational cost O(L × T) for sequences of length T.

1 Introduction: LTCs define hidden-state flow with linear decay and gated input currents: dx(t)/dt = −x(t)/τ + f(x(t), I(t), t, θ)(A−x(t)).The nonlinear gate modulates the interaction between the hidden state and parameter vector A.
1 Introduction: The gate creates an input-dependent liquid time-constant, τsys = τ/(1+τf(x(t),I(t),t,θ)), allowing hidden-state elements to specialize to arriving input features.LTCs can be implemented with an arbitrary choice of ODE solver.
1 Introduction: LTC states and time-constants are bounded to finite ranges, supporting stable output dynamics even when inputs relentlessly increase.The paper also presents a theoretical analysis of LTC approximation capability and expressivity.
1 Introduction: LTCs are inspired by steady-state synaptic-current models in which a sigmoidal nonlinearity multiplies the difference between a bias-like vector and neuronal voltage.The cited biological foundation yields an equation similar to the LTC formulation.
2 LTCs forward-pass by a fused ODE solvers: Because LTC equations are nonlinear and stiff, Runge-Kutta integrators such as Dormand–Prince are unsuitable, motivating a solver that fuses explicit and implicit Euler methods.The fused method replaces linearly occurring x(t_i) terms with x(t_i+1), allowing the next state to be solved symbolically.
2 LTCs forward-pass by a fused ODE solvers: The fused solver discretizes the continuous interval into steps and updates the neuronal state from t_i to t_i+1, producing one LTC update state per step.Algorithm 1 specifies parameters including τ, recurrent weights, biases, the step size Δt, and the number of unfolding steps L.
2 LTCs forward-pass by a fused ODE solvers: O(L × T) is the computational complexity of the algorithm for an input sequence of length T, where L is the number of discretization steps.The network can use an arbitrary activation function, such as tanh in f.

Training LTC networks by BPTT · 4 Bounds on τ and neural state of LTCs · 5 On the expressive power of LTCs

The paper trains LTCs with direct BPTT to accurately recover forward trajectories, establishes finite bounds on their time-constants and hidden states, and analyzes their expressive power through universality and trajectory-length growth. Theoretical and experimental results characterize LTCs as stable, expressive continuous-time models.

Training LTC networks by BPTT: The adjoint sensitivity method has constant memory cost per layer but introduces numerical errors because it forgets forward-time computational trajectories.This limitation motivates direct BPTT for LTC training.
Training LTC networks by BPTT: Direct BPTT trades memory for accurate forward-pass recovery during reverse-mode integration and trains solver outputs folded recursively into an RNN.The method is designed to maintain a highly accurate backward-pass integration through the ODE solver.
4 Bounds on τ and neural state of LTCs: LTC time-constants and hidden states are bounded to finite ranges, supporting stable behavior even with unbounded arriving inputs.The paper formalizes these properties in Theorems 1 and 2.
4 Bounds on τ and neural state of LTCs: The hidden-state bound guarantees that LTC outputs never explode even when inputs grow to infinity.The proof uses the signs of the LTC equation’s compartments and an explicit Euler approximation.
5 On the expressive power of LTCs: LTCs are universal approximators for bounded rollouts of autonomous C1 dynamical systems on compact initial-value sets.The construction embeds the target n-dimensional dynamical system into a higher-dimensional LTC system with proper initialization.
5 On the expressive power of LTCs: LTC universality is non-trivial because the network includes a nonlinear input-dependent term in its time-constant module.This distinguishes the proof from the universality proof for CT-RNNs.
5.1 Measuring expressivity by trajectory length: Trajectory length measures relative complexity by projecting hidden activations onto their first two principal components and measuring the resulting 2D latent trajectory.Experiments use circular inputs I(t) = {I1(t) = sin(t), I2(t) = cos(t)} over t ∈[0, 2π], with average variance explained over 80%.
5.2 Discussion of the theoretical bounds: LTC trajectory length grows faster than linearly, while CT-RNNs have shorter trajectories than Neural ODEs according to the theoretical exponent base.The faster-than-linear LTC growth is shown in Fig. 2B and Fig. 3C and matches the reported experiments.

6 Experimental Evaluation · 7 Related Works · 8 Conclusions, Scope and Limitations

Experiments report that LTCs outperform several recurrent and continuous-time baselines across diverse supervised time-series tasks, while related work situates their trajectory-length expressivity analysis among existing measures. The conclusion emphasizes effective solver implementation, stable and bounded dynamics, superior expressivity, strong prediction performance, and causality as future work.

6 Experimental Evaluation: LTCs were evaluated against discretized RNNs, LSTMs, CT-RNNs, CT-GRUs, and Neural ODEs across diverse real-life supervised learning tasks.The Neural ODEs used a fourth-order Runge–Kutta solver.
6 Experimental Evaluation: In human-activity prediction, LTCs outperformed all models in the first setting, particularly CT-RNNs and Neural ODEs, by a large margin.The dataset contains 6554 activity sequences sampled with a 211 ms period.
6 Experimental Evaluation: Under a second setting matched to prior modifications, LTCs showed superior performance by a high margin against a broader set of RNN variants.This setup was designed to provide a fair comparison with models discussed in prior work.
6 Experimental Evaluation: In Half-Cheetah autoregressive modeling, LTCs showed superior performance compared to other models despite randomly overwriting 5% of actions.The task fit observation-space time series generated by the MuJoCo physics engine.
7 Related Works: Time-continuous networks are linked to adaptive computation, continuous-time series modeling, memory, and parameter efficiency, alongside work on adjoint methods and specialized Neural ODE applications.The related-work discussion presents these as motivations for the growing popularity of continuous-time models.
7 Related Works: The paper extends trajectory-length expressivity analysis from static networks to time-continuous networks, beyond approaches tied to particular weight configurations.Earlier studies used linear-region counts, radial-function separation, transient chaos, or trajectory length to characterize expressivity.
8 Conclusions, Scope and Limitations: The authors report that LTCs combine linear ODE neurons with nonlinear weight configurations, support arbitrary variable- and fixed-step solvers, train through backpropagation through time, and exhibit bounded, stable dynamics.They also report superior expressivity and supervised time-series prediction performance relative to standard approaches.
8 Conclusions, Scope and Limitations: Studying causality in performant recurrent models such as LTCs is identified as an important future direction because continuous-time recurrent semantics resemble dynamic causal models.The models map past experiences to next-step predictions through recurrent mechanisms.

Supplementary Materials · S1 Proof of Theorem 1

The proof bounds the system dynamics by replacing the bounded sigmoidal nonlinearity with its upper and lower limits. These substitutions reduce the equations to linear ordinary differential equations, yielding lower and upper bounds on the system’s time constant.

S1 Proof of Theorem 1: The proof assumes f is a bounded, monotonically increasing sigmoidal nonlinearity whose values lie between 0 and 1.This assumption is applied neuron-by-neuron.
S1 Proof of Theorem 1: Replacing the upper bound of f and assuming a scaling weight matrix W M×1 produces the first proof simplification.The passage specifies the scaling weight matrix as W M×1.
S1 Proof of Theorem 1: The resulting equation simplifies to a linear ODE.This is the first reduced dynamical form used in the proof.
S1 Proof of Theorem 1: The neuron-wise expression is obtained for each neuron i in f.The passage presents the per-neuron form following the upper-bound substitution.
S1 Proof of Theorem 1: The linear ODE has an explicit solution, from which the lower bound of the system’s time constant, τ min, is derived.The proof first states the solution form and then derives τ min.
S1 Proof of Theorem 1: Replacing the lower bound of f simplifies the equation to an autonomous linear ODE and gives the upper bound of the system’s time constant, τ max.The lower-bound substitution establishes the complementary time-constant bound.

S2 Proof of Theorem 2

The proof develops Theorem 2 by substituting the neuron state into the governing equation, expanding the resulting brackets, and deriving bounds through sign analysis and analogous upper-bound reasoning.

The proof substitutes x_i(t) as neuron i’s neural state into Equation 1.
Expanding the brackets produces the next proof expression.
The right-hand side is negative under the stated conditions, so the left-hand side is negative after approximating the derivative term.
Substituting x_i(t) with M provides the next bound expression.
A similar methodology to the upper-bound derivation yields the corresponding result.

S3 Proof of Theorem 3

The proof establishes that LTCs can approximate any finite-time trajectory of an n-dimensional autonomous dynamical system. It combines universal approximation with boundedness, uniqueness, and an explicit LTC construction using recurrent hidden states and feed-forward outputs.

Theorem 3: LTC internal and output states can approximate any n-dimensional dynamical system over a finite simulation interval with suitable hidden nodes and initialization.The argument builds on universal approximation results for feedforward and recurrent neural networks.
Approximation foundations: The proof uses the fundamental approximation theorem, which guarantees approximation of continuous mappings on compact sets by three-layer feedforward networks.The theorem assumes a continuous target mapping on a compact subset of R^n and permits arbitrary positive approximation error.
Existence and boundedness: An LTC defined by Eq. S20 has a unique solution on [0, ∞) because its output function is bounded and C1.The surrounding lemmas establish local Lipschitzness, boundedness, and global existence conditions for the differential equations.
LTC construction: The construction defines a C1 dynamical mapping with LTC-matched parameters, then selects system time-constants and weights so its trajectories approximate the target system.The proof introduces ˜F(x) = −(1/τ + W_lf(γx + µ))x + W_lf(γx + µ)A and matches W_l to W.
Network structure and scope: The constructed system is realized by recurrent hidden states and output states, with hidden-to-hidden recurrence but feed-forward connections from hidden nodes to outputs.The proof assumes no inputs and shows that hidden and output units together approximate finite trajectories of an autonomous dynamical system.

S4 Proof of Theorem 4

The section proves a trajectory-length lower bound for Neural ODEs and begins extending the derivation to CT-RNNs. The proof uses latent trajectory representations, weight decompositions, activation saturation, and Gaussian norm results.

Proof of Theorem 4: The analysis measures expressivity through trajectory length in latent trajectory space, following the approach introduced for static deep neural networks.The section motivates adapting this measure to continuous-time models with careful treatment of their continuous dynamics.
Neural ODE lower bound: Weight-matrix decompositions into parallel and perpendicular components, together with independent Gaussian projections, drive the recurrence bounding latent-state gradients.The proof conditions on perpendicular input components and uses projection and Gaussian norm lemmas to control successive-layer contributions.
Proof of Theorem 4: The proof first establishes the lower bound for Neural ODEs and then extends the result to CT-RNNs.The Neural ODE derivation proceeds from gradients between successive hidden states and their first two principal components.
Neural ODE lower bound: For Neural ODEs, the asymptotic bound is obtained for input trajectories orthogonal to their successive time-points, with c1 ≈1.The resulting infinitesimal hidden-state bound is then accumulated across L ODE steps to bound the overall trajectory length.
CT-RNN lower bound: The CT-RNN extension starts from its continuous-time state gradient, where the time-constant vector τ is conditioned to be strictly positive.An explicit Euler discretization is compared with the corresponding Neural ODE discretization to continue the trajectory-length derivation.

Proof of Theorem 5

The proof derives the latent-state gradient dynamics for successive LTC layers, then bounds parallel and orthogonal contributions using parameter assumptions, Hard-tanh saturation, and recursive unrolling. The resulting asymptotic bounds establish Theorem 5 for finite input sequences.

Gradient dynamics: The proof starts from the gradient between successive states in layer d+1 and transfers the analysis to the first two principal components of the latent representation.The latent dynamics use the same recurrent structure, with h^(d) computed from the preceding representation.
Gradient dynamics: The norm expectation is decomposed into parallel and orthogonal weight-matrix components, with conditioning used to separate expectations of products.The derivation applies Lemma 9 and conditions on the weight matrix during the norm analysis.
Activation assumptions: For Hard-tanh activations, the non-saturated index set controls the contribution of time-constant weights, whose average effect enters as a constant addition.The cardinality |A| counts non-saturated states and determines their average contribution.
Asymptotic bounds: The proof obtains asymptotic bounds separately for the right-hand expression and the perpendicular component of the hidden-state gradient.The bounds use Gaussian weight assumptions and established bounds for deep Hard-tanh networks.
Theorem conclusion: Recursive unrolling through the n-layer network extends the bound to an input sequence of length T, yielding Equation S120 and the theorem statement.The recursion proceeds layer by layer from the latent state back to the input.

S5 Experimental Setup - Section 6

The experiments compare recurrent models using standardized training, validation, testing, and evaluation procedures across classification, segmentation, and forecasting tasks. The setup specifies 32-unit RNNs, task-specific output projections, fixed-step ODE solvers, and dataset-specific sequence preprocessing.

General Protocol: Models were trained with dataset-dependent 75:10:15 training-validation-test splits, checkpointed by validation performance, restored, tested, and evaluated across five repetitions.The stated split ratios were approximate because exact ratios depended on the dataset.
Model Configuration: Each RNN used 32 hidden units and a learnable linear output layer, with experiments targeting comparative expressive power and generalization rather than peak predictive performance.The output projection accommodated task-specific output dimensions.
ODE Solvers: CT-RNNs, Neural ODEs, and LTCs used fixed-step Euler, fourth-order Runge-Kutta, and fused ODE solvers, respectively, with six solver steps per RNN step.The solver time-step was set to one-sixth of the input sampling frequency.
Classification and Segmentation: Classification and segmentation tasks covered hand gestures, room occupancy, human activities, and sequential MNIST using sensor, inertial, or image-derived sequences.Hand gestures used 32 sensor features and five labels; human activity recognition used 561 features and six categories; sequential MNIST encoded samples as 28-dimensional sequences of length 28.
Forecasting: Forecasting tasks predicted hourly highway traffic, household active power consumption, and ozone days from weather, sensor, temporal, and solar-radiation features.The ozone task used overlapping 32-day sequences and labeled a day positive when ozone exceeded 80 parts per billion for at least 8 hours.
Person Activity - 1st Setting: A separate Person Activity setting used 25 recordings, four sensors sampled every 211 ms, seven packed classes, unnormalized inputs, and partially overlapping 32-step subsequences.Its results were not directly transferable to the cited prior study because the split seeds and input representation differed.

S6 Hyperparameters and Parameter counts - Tables 3, 4, and 6

This section reports the hyperparameters used in the experimental evaluations and compares parameter counts across RNN architectures as functions of width, hidden layers, and decay slots.

Hyperparameters: Table S1 lists the hyperparameters used for the experimental evaluations.The passage identifies the table but provides no individual hyperparameter values.
Overview: Together, the tables document experimental settings and model-size scaling variables used in the evaluations.This summarizes the two table topics without adding numerical or comparative claims.
Parameter counts: Table S2 gives parameter counts for various RNN models in relation to RNN width k, hidden-layer count n, and decay-slot count m.The passage specifies the governing dimensions but provides no individual parameter-count values.

S7 Additional trajectory space representations:

Additional trajectory-space representations for the reported results are available for viewing online.

S7 Additional trajectory space representations: Additional trajectory space representations for the results are provided.The representations can be viewed online.
S7 Additional trajectory space representations: The reported results include a trajectory space representation resource.The passage directs readers to an online archive.
S7 Additional trajectory space representations: The additional LTC trajectory representations are accessible through the cited Dropbox archive.The archive is linked directly in the passage.

S8 Trajectory Length results

This section presents additional trajectory length results, shown in Figure S1.

S8 Trajectory Length results: Figure S1 reports additional trajectory length results.
S8 Trajectory Length results: The reported results concern trajectory length.
S8 Trajectory Length results: The figure is identified as Figure S1.