PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Long-Term Dependencies in Irregularly-Sampled Time Series

Mathias Lechner, Ramin Hasani

arXiv:2006.04418v4cs.LGstat.ML

TL;DR

Irregularly sampled time series are well suited to continuous-time RNNs, but long-term dependencies expose vanishing or exploding gradients. The paper proves this issue persists across ODE solvers and introduces ODE-LSTMs, which separate LSTM memory from a continuous-time output state; experiments show consistently better performance than standard and advanced continuous-time RNN variants.

  • Problem

    Continuous-time RNNs model irregularly sampled data but struggle to learn long-term dependencies because of vanishing or exploding gradients.

  • Method

    ODE-LSTMs preserve LSTM memory while embedding continuous-time dynamics in the output state, allowing arbitrary time-lags and constant error propagation through memory.

  • Results

    ODE-LSTMs consistently outperform standard and advanced continuous-time RNN variants on synthetic and real-world sparse time-series tasks.

  • Takeaways & Limitations

    ODE-LSTMs provide a continuous-time RNN design for learning long-term dependencies in irregularly sampled input streams.

  • Takeaways & Limitations

    Augmented RNNs rely on learning time behavior from the data and may generalize uncertainly to unseen time-lags; experiments found them outperformed by explicitly continuous-time models.

Abstract

from arXiv · show

Recurrent neural networks (RNNs) with continuous-time hidden states are a natural fit for modeling irregularly-sampled time series. These models, however, face difficulties when the input data possess long-term dependencies. We prove that similar to standard RNNs, the underlying reason for this issue is the vanishing or exploding of the gradient during training. This phenomenon is expressed by the ordinary differential equation (ODE) representation of the hidden state, regardless of the ODE solver's choice. We provide a solution by designing a new algorithm based on the long short-term memory (LSTM) that separates its memory from its time-continuous state. This way, we encode a continuous-time dynamical flow within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path. We call these RNN models ODE-LSTMs. We experimentally show that ODE-LSTMs outperform advanced RNN-based counterparts on non-uniformly sampled data with long-term dependencies. All code and data is available at https://github.com/mlech26l/ode-lstms.

1 Introduction

ODE-RNNs naturally model irregularly sampled time series but suffer from vanishing or exploding gradients when learning long-term dependencies. ODE-LSTMs combine continuous-time dynamics with LSTM memory and outperform standard and advanced continuous-time RNN variants on sparse time-series tasks.

  • Motivation: ODE-RNNs model irregularly sampled data through ordinary differential equations but provably suffer from vanishing and exploding gradients during reverse-mode training.The issue also persists when ODE-RNNs are trained with the adjoint method.
  • Motivation: LSTMs preserve constant error propagation and learn long-term dependencies, but discrete-state updates struggle when observations arrive at irregular time gaps.Their memory and output states are separated, yet their discretized dynamics remain mismatched to irregular sampling.
  • Approach: ODE-LSTMs feed continuous-time output states back into LSTM gates, enabling responses to arbitrary time-lags while retaining the memory mechanism.The architecture embeds a continuous-time dynamical flow within the LSTM cell.
  • Results: ODE-LSTMs consistently outperform standard and advanced continuous-time RNN variants across synthetic and real-world sparse time-series tasks.The comparison covers tasks involving irregularly sampled data.

3 ODE-RNNs suffer from vanishing or exploding gradient

The paper analyzes gradient propagation in ODE-RNNs and shows that vanishing or exploding gradients arise from their continuous-time dynamics, independently of common solver choices. Only degenerate constant dynamics avoid the problem.

  • Gradient propagation: Gradient propagation in recurrent networks is governed by repeated state Jacobians, whose magnitude determines whether errors vanish, explode, or remain constant.The error-flow structure directly affects the effectiveness of learning.
  • Solver-based analysis: ODE-RNNs trained with explicit Euler or Runge-Kutta methods suffer from vanishing or exploding gradients except under non-trainable or sequence-specific constant dynamics.This is stated formally as Theorem 1.
  • Solver-independent result: The vanishing or exploding gradient persists regardless of the ODE solver, apart from degenerate constant-dynamics configurations.Theorem 2 applies to the ODE solution under a uniformly Lipschitz continuous vector field.
  • Solver-based analysis: Refining the discretization grid does not resolve the gradient-propagation issue, because the discrete approximations converge to the same problematic ODE solution.The argument uses Picard–Lindelöf convergence and bounds on the limiting gradients.
  • Adjoint training: The adjoint method does not eliminate the gradient problem because lossy reverse integration diverges from the forward trajectory.Memory efficiency therefore does not make adjoint training immune to gradient issues.

4 ODE-LSTM architecture

ODE-LSTM preserves the LSTM memory cell while making its output state continuous in time. This separates long-term error propagation from time-dependent dynamics for irregularly sampled inputs.

  • Architecture: ODE-LSTM retains the LSTM memory cell for constant error propagation and modifies the output state to represent continuous-time dynamics.The memory cell carries long-term information while the output state models nonlinear, time-dependent behavior.
  • Architecture: The continuous-time output state makes the gating dynamics depend on elapsed time when processing irregularly sampled sequences.ODE-RNN dynamics are used to define the cell’s output between observations.
  • Design distinction: Unlike decaying-memory variants, ODE-LSTM leaves the memory mechanism untouched and places continuous dynamics in the output function.This design is intended to support learning long-term dependencies by gradient descent.
  • Gradient behavior: With suitable near-zero initialization, ODE-LSTM memory units avoid vanishing or exploding gradients at the beginning of training.Theorem 3 establishes this property for the memory units under the stated initialization assumptions.
  • Gradient behavior: The forget-gate bias controls the initial error-flow factor, allowing it to be moved closer to 1 for very long-term dependencies.The cited example gives σ(1) ≈ 0.7310586, which is between zero and one.
  • Architecture: ODE-LSTM is a memory cell whose gates are controlled by a time-continuous ODE process.The architecture is evaluated on multiple time-series prediction tasks.

5 Experimental evaluation

The evaluation tests ODE-LSTM on synthetic and real-world irregularly sampled tasks designed to expose long-term dependencies. Across bit-stream classification, person activity recognition, event-based sequential MNIST, and Walker2d modeling, ODE-LSTM generally outperforms competing RNN variants.

  • Experimental setup: The experiments evaluate continuous-time RNNs on synthetic and real-world sparse time-series tasks with deliberately long-term dependencies.Baselines include continuous-state, decay-based, oscillatory, augmented, and bidirectional RNN variants.
  • Bit-stream sequence classification: ODE-LSTM outperformed other models in event-based bit-stream encoding, while several ODE-based variants failed to solve the XOR task in dense encoding.ODE-RNNs, CT-RNNs, RNN-Decay, Phased-LSTM, and GRU-ODE could not solve the first modality; ODE-LSTM performed best in the event-based modality.
  • Person activity recognition: ODE-LSTM outperformed other RNN models on the irregularly sampled person activity recognition dataset.The task provides an error signal at every time-step, making vanishing gradients less central than in sequence-level classification.
  • Event-based sequential MNIST: ODE-LSTM surpassed other models on event-based sequential MNIST, whereas ODE-RNN, CT-RNN, and GRU-ODE struggled to learn a high-fidelity model.The compressed sequences contain 256 irregularly sampled datapoints, requiring information storage across long time-spans and time-lag handling.
  • Walker2d kinematic simulation: ODE-LSTM captured Walker2d kinematic dynamics better than other algorithms with a high margin on an irregularly sampled autoregressive modeling task.The dataset uses simulated MuJoCo dynamics, randomized actions, and frame-skips that remove 10% of time-steps.

6 Discussions, Scope and Limitations

The paper contrasts time-lag augmentation with architectures that explicitly model continuous time, and reports that ODE-LSTMs outperform a bidirectional LSTM-ODE-RNN design.

  • 6 Discussions, Scope and Limitations: Augmented LSTM feeds the samples’ time-lag as an additional input feature, leaving the network to learn how elapsed time should affect its state.This makes irregular-time handling depend on the learning process.
  • 6 Discussions, Scope and Limitations: ODE-LSTMs explicitly declare their state in continuous time and outperform the augmented LSTM in the reported experiments.
  • 6 Discussions, Scope and Limitations: A bidirectional LSTM-ODE-RNN could combine irregular-time modeling with long-term dependency learning, but heterogeneous learning speeds may limit training.
  • 6 Discussions, Scope and Limitations: ODE-LSTMs interlink LSTM and ODE-RNN components architecturally rather than autoregressively, and consistently outperform the bidirectional LSTM-ODE-RNN.

7 Related Works

The related-work discussion situates ODE-LSTMs among continuous-time and long-memory RNNs, emphasizing the challenge of learning long dependencies in irregularly sampled data and the proposed solution.

  • Time-continuous RNNs: CT-RNNs add a dampening factor, while GRU-D models time lags through a trainable decaying mechanism and achieved competitive performance in the experiments.
  • Time-continuous RNNs: CT-GRU adds multiple decay factors and attention over decay dimensions, but the paper reports that this mechanism fails in Table 2.
  • Time-continuous RNNs: Phased-LSTM uses a learnable oscillator to model elapsed time but introduces a vanishing factor in its gradients.
  • Time-continuous RNNs: GRU-ODE makes error propagation time-dependent, abolishing GRU’s near-constant error-propagation property.
  • Time-continuous RNNs: CT-LSTM interpolates between two candidate hidden states according to elapsed time.
  • Learning Irregularly-Sampled Data: Continuous-time RNNs are a natural fit for non-uniformly spaced data, but ODE-RNN performance drops substantially when samples contain long-range dependencies.
  • Learning Long-term Dependencies: Vanishing and exploding gradients are identified as the core reason RNNs struggle with generalizability and long-term dependencies.
  • Learning Irregularly-Sampled Data: ODE-LSTMs address this shortcoming by enabling ODE-based RNNs to learn long-term dependencies in irregularly sampled inputs.

8 Conclusion

The paper concludes that ODE-LSTMs combine continuous-time dynamics with LSTM-style long-term memory for irregularly sampled streams. They resolve the gradient problem identified in ODE-RNNs and perform attractively on long-term dependency tasks.

  • 8 Conclusion: ODE-LSTMs use a continuous-time output state to modify the LSTM’s internal dynamical flow for irregularly sampled input streams.
  • 8 Conclusion: ODE-LSTMs resolve vanishing and exploding gradients in ODE-RNNs while performing attractively on long-term dependencies with non-uniform intervals.

Broader Impact

The paper frames ODE-LSTMs as relevant to non-uniform, missing-value time series in safety-critical domains, while noting the need for robustness analysis before deployment.

  • Broader Impact: Non-uniform intervals and missing values occur routinely in medical, business, social, and industrial automation settings.
  • Broader Impact: The reported results enable systems that handle irregular sampling while learning long-term dependencies important to their applications.
  • Broader Impact: Surgical and intensive-care applications require agents to use irregular process data and assign credit to past actions for current decisions.
  • Broader Impact: Deployment in high-stakes decision-making requires robustness analysis against perturbations, noise, and adversarial attacks.
  • Broader Impact: The method’s learning-to-forget and long-term-dependency mechanisms can help control data biases when designing fair machine-learning systems.

S1 Proofs

The proofs show that ODE-RNN gradients can vanish or explode regardless of discretization refinement or solver choice. ODE-LSTM instead uses an explicit memory path whose Jacobian supports near-constant error propagation under suitable initialization.

  • ODE-RNN gradient behavior: ODE-RNNs discretized with Euler or Runge-Kutta methods inherit error-flow terms that can depend on recurrent weights and time constants.The analysis covers both explicit Euler and Runge-Kutta discretizations, with Euler identified as a special Runge-Kutta case.
  • ODE-RNN gradient behavior: Depending on the recurrent derivative and time constant, the ODE-RNN error flow can either explode or vanish.A non-zero time constant pushes gradients toward a vanishing region, while other weight regimes can produce explosion.
  • ODE-LSTM solution: Constant error propagation requires an explicit memory element that remains unchanged between arbitrary time points.The proof identifies zero change in the ODE state as the only route to constant propagation, motivating a separate memory path.
  • ODE-RNN gradient behavior: Refining the discretization grid does not resolve the gradient-propagation problem, even though the numerical state converges to the true ODE solution.The proof invokes Picard–Lindelöf convergence while separately showing that gradient issues persist.

S2 Experimental evaluation

The evaluation uses irregularly sampled sequence benchmarks, including event-based bit streams, human activity, event-based seqMNIST, and Walker2d kinematics. These datasets vary in encoding, sampling irregularity, sequence construction, and evaluation setup.

  • Data encoding: Event-based encoding compresses sequences by recording changes rather than consecutive repeated symbols, producing irregularly sampled inputs.For binary sequences, repeated values are removed and time is added to preserve event timing.
  • Benchmarks: The evaluation includes event-based bit-stream classification, human activity recognition, event-based seqMNIST, and Walker2d kinematic modeling.Walker2d predicts the next environment state, while the activity task performs per-time-step classification.
  • Benchmark construction: The XOR benchmark uses 32-bit random blocks, parity-based binary labels, 100,000 training samples, and 10,000 test samples.Training uses cross-entropy loss, and the training set covers less than 0.0024% of all possible bit streams.
  • Benchmark construction: The human-activity data contain 25 recordings from inertial sensors with phase shifts that make the series irregularly sampled.The task groups eleven activities into seven classes and trains on partially overlapping sequences of length 32.
  • Evaluation scope: The reported human-activity results are not directly comparable with prior work because the input features use a different representation.The paper uses a seven-dimensional feature vector rather than the prior value-mask representation.
  • Benchmark construction: Walker2d irregularity is introduced by removing 10% of time steps, while the benchmark uses 300 training, 40 validation, and 60 test rollouts.The model predicts the next kinematic state, and mean-square error is used for training and evaluation.
Loading 2006.04418v4…