Source-linked AI summary
Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
Daniel Neil, Michael Pfeiffer, Shih-Chii Liu
TL;DR
Irregularly sampled and event-driven sequences challenge conventional fixed-step RNNs, especially when sensors operate at different rates. Phased LSTM adds a rhythmic time gate that permits sparse, asynchronous updates; across long-sequence, sensory, and standard tasks, it converges faster while often improving accuracy and reducing runtime computation.
Problem
Conventional fixed-step RNNs are ill-suited to irregularly sampled event streams and inputs from sensors with different update rates.
Method
Phased LSTM extends LSTM with a learnable rhythmic time gate that permits cell and hidden-state updates only during brief open phases and supports irregular update times.
Results
Across the presented experiments, Phased LSTM converges more quickly, often improves accuracy over standard LSTM, and theoretically requires only 5% of runtime computes.
Takeaways & Limitations
The model supports asynchronous sensory-event processing while retaining timing information and improving LSTM performance in standard RNN applications.
Abstract
from arXiv · showhide
Recurrent Neural Networks (RNNs) have become the state-of-the-art choice for extracting patterns from temporal sequences. However, current RNN models are ill-suited to process irregularly sampled data triggered by events generated in continuous time by sensors or other neurons. Such data can occur, for example, when the input comes from novel event-driven artificial sensors that generate sparse, asynchronous streams of events or from multiple conventional sensors with different update intervals. In this work, we introduce the Phased LSTM model, which extends the LSTM unit by adding a new time gate. This gate is controlled by a parametrized oscillation with a frequency range that produces updates of the memory cell only during a small percentage of the cycle. Even with the sparse updates imposed by the oscillation, the Phased LSTM network achieves faster convergence than regular LSTMs on tasks which require learning of long sequences. The model naturally integrates inputs from sensors of arbitrary sampling rates, thereby opening new areas of investigation for processing asynchronous sensory events that carry timing information. It also greatly improves the performance of LSTMs in standard RNN applications, and does so with an order-of-magnitude fewer computes at runtime.
1 Introduction
Fixed update rates constrain precision and efficiency when sensors operate at different sampling rates or generate asynchronous event streams. The paper presents an RNN model designed to process inputs sampled at asynchronous times.
- Different sensors may require different sampling rates, forcing short time steps that increase computational load and power consumption.
- Event-based sensors produce sparse, asynchronous streams with short latencies and precise timing, but time-stepped models would require high update frequencies.
- Biological neurons communicate with spikes and perform asynchronous, event-triggered updates in continuous time.
- The work introduces a novel RNN model that processes inputs sampled at asynchronous times.
2 Model Description
Phased LSTM adds a learnable rhythmic time gate to LSTM, allowing irregular updates, sparse computation, and longer memory retention. The gate controls when cell and hidden states update while preserving state during closed phases.
- Phased LSTM extends LSTM with a time gate controlled by a learnable rhythmic oscillation with period, open-ratio, and phase-shift parameters.The period τ controls real-time cycle length, ron controls the open-phase fraction, and s controls phase shift.
- Cell and hidden states update only during the gate’s open phase; during the closed phase, the previous state is maintained.In multilayer networks, timestamps are distributed to all layers, which update at the same time point.
- The linearized time gate rises and falls during open phases, remains closed during a third phase, and uses leak rate α to transmit gradient information while closed.The linear slopes during open phases support effective error-gradient transmission.
- Phased LSTM can update at irregularly sampled times, enabling operation on event-driven, asynchronously sampled input data.Its cell and hidden updates are mediated by the time gate at update times tj.
- During closed phases, the Phased LSTM state can maintain perfect memory, while updates occur for only ron · τ within each oscillation period.This cyclic memory provides a longer, adjustable memory length through τ.
- Sparse updates reduce runtime updates and shorten the sequence of updating timesteps through which gradients must backpropagate.The time gate also prevents external inputs and time steps from dispersing and mixing the cell-state gradient.
3 Results
Across timing-sensitive, long-sequence, event-based, and multimodal tasks, Phased LSTM converges faster and handles irregular sampling while reducing runtime updates. Its benefits appear in asynchronous frequency discrimination, long addition sequences, N-MNIST recognition, and lip reading.
- Frequency Discrimination Task: Phased LSTM handles standard, oversampled, and asynchronously sampled sine-wave inputs, whereas LSTM and BN-LSTM degrade under oversampling and asynchronous timing.The Phased LSTM’s time gates can be sampled at any continuous time within the oscillation period.
- Frequency Discrimination Task: The Phased LSTM converges quickly and exhibits low training variance on the more challenging superimposed-frequencies task.Figure 4a compares the same network parameters across Phased LSTM, LSTM, and BN-LSTM.
- Adding Task: Longer time-gate periods accelerate training on the adding task, which requires retaining two marked values across sequences of roughly 500 numbers.The longer period provides a longer jump through past timesteps during backpropagation-through-time.
- N-MNIST Event-Based Visual Recognition: On N-MNIST, Phased LSTM trains faster, reaches higher accuracy with lower variance, and remains effective when event inclusion changes without retraining.The model processes spatio-temporal event streams directly rather than relying on blurred frame-based representations.
- Visual-Auditory Sensor Fusion for Lip Reading: In lip reading, Phased LSTM converges substantially faster than LSTM and BN-LSTM, with peak accuracy of 81.15% under the evaluated video-only conditions.Higher audio sampling provided 10 ms rather than 40 ms latency without increasing accuracy.
4 Discussion
Phased LSTM’s rhythmic gating supports fine timing discrimination, persistent state, and shortcuts to the past for gradient backpropagation. Across experiments, it converges faster, theoretically uses 5% of standard runtime computes, and often improves accuracy.
- Phased LSTM’s rhythmic periodicity permits fine timing discrimination by acting like a learnable, gated Fourier transform on its input.
- The rhythmic periodicity can function as persistent dropout that preserves state and enhances model diversity.
- Rhythmic inactivation provides a shortcut to the past for gradient backpropagation, accelerating training.
- 5% of the computes at runtime are theoretically required by Phased LSTM, while accuracy often improves compared to standard LSTM.
- The presented results demonstrate discrimination of rhythmic signals and learning of long memory traces.
- The method can be extended to GRUs, and simpler square-wave-like oscillations may support more efficient formulations.