Source-linked AI summary
Can recurrent neural networks warp time?
Corentin Tallec, Yann Ollivier
TL;DR
The paper asks how recurrent models can remain effective when input timing changes, given ordinary RNNs’ sensitivity to time rescaling. It derives gating from time-warping invariance, introduces chrono gate-bias initialization, and reports benefits for long-term dependencies while identifying a discrete-time speed limitation.
Problem
Ordinary recurrent networks are nonresilient to time rescaling, so temporal tasks can become arbitrarily difficult when spacing between inputs changes.
Method
The paper derives learnable gating by requiring quasi-invariance to time warpings, then uses the resulting time-constant interpretation to prescribe chrono initialization.
Results
Chrono initialization heavily outperforms standard LSTM initialization on synthetic tasks and outperforms or competes with it on real-world problems.
Takeaways & Limitations
Learnable gates can regulate local time constants, providing a principled connection between time-warping invariance and recurrent architectures such as LSTMs and GRUs.
Takeaways & Limitations
The discrete-time analogy applies only when the time-warping derivative is not too large; gated models do not handle phenomena occurring too quickly for the model.
Abstract
from arXiv · showhide
Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use ad hoc gating mechanisms. Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues. We prove that learnable gates in a recurrent model formally provide quasi- invariance to general time transformations in the input data. We recover part of the LSTM architecture from a simple axiomatic approach. This result leads to a new way of initializing gate biases in LSTMs and GRUs. Ex- perimentally, this new chrono initialization is shown to greatly improve learning of long term dependencies, with minimal implementation effort.
1 FROM TIME WARPING INVARIANCE TO GATING
Ordinary recurrent networks are not invariant to changes in temporal scale, whereas learnable time-dependent gating yields quasi-invariance to sufficiently representable time warpings. This perspective derives a basic gated recurrent architecture and explains its connection to LSTMs and GRUs.
- Changing the time scale can make a task arbitrarily difficult for ordinary recurrent networks, including by inserting blanks between sequence elements.Their representable function class is not invariant to time rescaling.
- A learnable leak coefficient makes recurrent models invariant to time rescaling and represents the problem’s global characteristic timescale.The reciprocal coefficient is interpreted as the network’s characteristic forgetting time.
- General time-warping invariance requires a learnable function representing the derivative of the warping, with the warping learned from inputs seen so far.The warping may describe accelerations or decelerations as well as rescalings.
- The resulting gated recurrent models are quasi-invariant: their quality depends on how broad a class of time warpings the learnable gating function can represent.A universal approximator would provide perfect invariance, while a finite model provides only quasi-invariance.
- The simplest discrete model uses input gating and complementary forget gating, closely resembling LSTM cell and GRU hidden-unit updates.Per-unit gates additionally allow different local contraction or dilation functions for multiple timescales.
- Discrete-time gated models handle time warpings that stretch time, but their continuous-to-discrete approximation breaks down when changes occur too quickly for the model.The Taylor approximation is expected to hold when α ≲ 1 or g_t ≲ 1.
2 TIME WARPINGS AND GATE INITIALIZATION
The paper proposes initializing recurrent-network gate biases from the expected temporal-dependency range, linking gate values to memory timescales. For LSTMs, this yields the chrono initialization, whose biases are sampled from a range determined by T_min and T_max.
- Gate initialization: Gate biases control the order of magnitude of gate values over time and therefore influence the network’s memory range.When inputs and hidden layers are centered, the biases determine the typical gate-value scale.
- Gate initialization: The bias range −log(T_max −1) to −log(T_min −1) is a loose initial prescription, allowing later adaptation to rare changes in temporal scale.The authors recommend using these values only for initialization rather than constraining the gates permanently.
- Gate initialization: For dependencies in [T_min, T_max], initialize gate biases as −log(U([T_min, T_max]) −1), using U as the uniform distribution.The prescription is intended to match the model’s memory timescale to the data’s characteristic timescales.
- LSTM specialization: In LSTMs, the recurrent-model gate g corresponds to input gate i_t, while 1 − g_t corresponds to forget gate f_t.This correspondence connects the general gated recurrence to the LSTM cell update.
- LSTM specialization: The resulting chrono initialization is a principled LSTM gate-bias initialization based on the expected temporal range of the sequential data.The section identifies this procedure as the chrono initialization.
3 EXPERIMENTS
Experiments test whether gating improves robustness to time warping and whether chrono initialization improves learning of long-term dependencies. Gated RNNs remain robust across warping conditions, while chrono initialization substantially improves several long-memory tasks.
- Experimental scope: The experiments evaluate synthetic long-term-dependency tasks alongside Text8 and Penn Treebank next-character or next-word prediction.The supplied experimental description identifies both synthetic test suites and real-world language-modeling datasets.
- Pure warpings and paddings: The pure-warping task isolates robustness to temporal distortion by requiring prediction of the previous character in randomly warped sequences.Variable warping repeats each character a random number of times, using the same warping for input and output.
- Pure warpings and paddings: Gated RNNs achieve perfect performance across uniform and variable time-warping setups, whereas leaky RNNs fail on variable warpings.Leaky RNNs solve uniform warpings but are not invariant to variable warpings; gated architectures significantly outperform ordinary RNNs with moderate warping.
- Chrono initialization: Chrono initialization largely outperforms standard initialization on copy tasks, including variable copy at T=1000.For variable copy at T=500, both initializations perform similarly, while chrono initialization is substantially better at T=1000.
- Chrono initialization: Chrono initialization significantly speeds learning on adding tasks and converges 7 times faster for T=750.The comparison uses 128-unit LSTMs with standard forget biases set to 1 and chrono initialization using T_max = T.
CONCLUSION
The paper derives recurrent gating from invariance to time warpings and introduces chrono initialization from the resulting time-constant interpretation. Experiments report notable benefits for long-term dependencies.
- Theoretical conclusion: Invariance to time warpings derives self-loop feedback gating from first principles and interprets gated connections as regulators of local time constants.The conclusion presents this as the theoretical basis for the architecture.
- Practical conclusion: Chrono initialization is presented as a principled method for initializing LSTM gate biases.The conclusion connects the initialization directly to the time-constant interpretation of gating.
- Empirical conclusion: Experiments show notable benefits from chrono initialization when learning long-term dependencies.The conclusion states the benefit without specifying a particular dataset or metric.
A ADDITIONAL EXPERIMENTS
Additional experiments evaluate recurrent architectures’ robustness to time warping and chrono initialization across synthetic, image, and language tasks. Chrono initialization improves permuted-MNIST performance and slightly improves text8 compression, while warping experiments reveal a trade-off between gated models’ accuracy and extrapolation behavior.
- Language modeling: Figure 7 compares standard and chrono initialization on word-level PTB and character-level text8 validation sets.The PTB experiment uses a deep recurrent highway network, while text8 uses next-character prediction.
- Generalization under time warping: Gated RNNs perform better overall but degrade faster beyond the training warp range, whereas plain RNNs remain below 80% in-range with flatter extrapolation.This contrasts higher in-range performance with more rapid degradation for gated models.
- Pixel-level classification: Chrono initialization raises permuted-MNIST validation accuracy from 95.4% to 96.3%, while non-permuted MNIST shows no clear difference.The experiments use LSTMs with 512 hidden units and set T_max to the input-sequence length, 784.