Source-linked AI summary
A Clockwork RNN
Jan Koutník, Klaus Greff, Faustino Gomez, Jürgen Schmidhuber
TL;DR
RNNs struggle to learn long-term dependencies in sequential data. This paper introduces CW-RNN, which runs hidden-layer modules at different clock speeds, and reports superior performance in preliminary audio generation and spoken-word classification experiments against SRN and LSTM.
Problem
Standard RNNs have difficulty learning long-term dependencies spanning more than 10 time-steps because of vanishing gradients.
Method
CW-RNN partitions the hidden layer into modules that process inputs at different discrete clock periods, executing only scheduled modules.
Results
CW-RNN outperformed both SRN and LSTM with the same number of weights in preliminary audio generation and TIMIT spoken-word classification experiments.
Takeaways & Limitations
Running subsets of neurons at different speeds allowed an RNN to efficiently learn different dynamic time-scales in complex signals.
Takeaways & Limitations
Initial experiments with giving each weight its own clock found those networks hard to train and unable to provide good results.
Abstract
from arXiv · showhide
Sequence prediction and classification are ubiquitous and challenging problems in machine learning that can require identifying complex dependencies between temporally distant inputs. Recurrent Neural Networks (RNNs) have the ability, in theory, to cope with these temporal dependencies by virtue of the short-term memory implemented by their recurrent (feedback) connections. However, in practice they are difficult to train successfully when the long-term memory is required. This paper introduces a simple, yet powerful modification to the standard RNN architecture, the Clockwork RNN (CW-RNN), in which the hidden layer is partitioned into separate modules, each processing inputs at its own temporal granularity, making computations only at its prescribed clock rate. Rather than making the standard RNN models more complex, CW-RNN reduces the number of RNN parameters, improves the performance significantly in the tasks tested, and speeds up the network evaluation. The network is demonstrated in preliminary experiments involving two tasks: audio signal generation and TIMIT spoken word classification, where it outperforms both RNN and LSTM networks.
1. Introduction
RNNs use recurrent connections for sequential problems but struggle to learn dependencies spanning more than 10 time-steps. CW-RNN addresses this by running hidden-layer modules at different clock speeds, reducing weights and evaluation cost while outperforming SRN and LSTM in preliminary tasks.
- Standard RNNs have difficulty learning long-term dependencies spanning more than 10 time-steps because of vanishing gradients.
- CW-RNN partitions the hidden layer into modules that run at different discrete clock periods.The architecture retains input, hidden, and output layers while assigning each module its own clock rate.
- CW-RNN reduces weights and speeds training and evaluation because slower modules are not connected to faster ones and not all modules execute every step.
- CW-RNN was evaluated on audio sequence generation and TIMIT spoken-word classification.
- In preliminary experiments, CW-RNN outperformed SRN and LSTM with the same number of weights by a significant margin.
2. Related Work
Related work addresses long time lags through specialized cells, optimization methods, additional delayed connections, leaky units, and multiscale recurrent structures. These approaches trade off trainability, parameters, computational cost, or adaptivity in different ways.
- The related-work focus is on RNN extensions for bridging long time lags.
- NARX RNN adds recurrent connections with delays of 2, 3, through k time steps, increasing parameters and making training more difficult.The additional connections also make execution k times slower.
- LSTM stores information indefinitely in a constant error carousel regulated by input, output, and forget gates.
- Reduced Description Networks and Echo State Networks use leaky neurons whose activations change only slightly in response to inputs.
- Hessian Free optimization helps RNNs solve some long-term-lag problems, but usually requires more optimization steps than LSTM.
- Hierarchical models can run RNNs at different time scales, but the Sequence Chunker, Neural History Compressor, and HTM require unsupervised event predictors.
3. A Clockwork Recurrent Neural Network
CW-RNN partitions hidden neurons into modules with distinct clock periods and structured recurrent connectivity. Only scheduled modules update at each time step, allowing slower modules to retain long-term information while faster modules process local detail and reducing computation.
- At t = 6, modules with periods T1 = 1 and T2 = 2 are evaluated, using corresponding contiguous block-rows of WH and WI.
- The hidden layer is partitioned into g internally fully connected modules, each assigned a clock period.Connections propagate from slower modules to faster modules when the faster module has the smaller period.
- At each time step, only modules satisfying (t MOD Ti) = 0 are executed; the paper uses exponential periods Ti = 2^(i−1).
- Non-executed modules retain their previous outputs, while executed modules update the corresponding output-vector components.
- Low-clock-rate modules retain long-term information, whereas high-speed modules focus on local high-frequency information with slower-module context available.
- The backward pass propagates error from executed modules and copies non-activated modules’ errors backward in time before adding them to the back-propagated error.
- The CW-RNN speedup has a lower bound of g/4 versus an RNN with the same number of neurons under the exponential-clock setup.
4. Experiments
The experiments compare similarly parameterized RNN, LSTM, and CW-RNN models on audio sequence generation and TIMIT spoken-word classification. CW-RNN performs best in both tasks, including a 5.7× lower error than LSTM on sequence generation and more than 2× lower error on classification.
- Experimental setup: Networks were compared with approximately equal parameter counts, using repeated random initializations and task-specific supervised training.The comparison included SRN, LSTM, and CW-RNN models.
- Sequence generation: The generation task required networks receiving no input to reproduce target audio sequences as accurately as possible.Five 320-point sequences sampled from music were used as targets.
- Sequence generation: CW-RNNs gave by far the best generation results, while RNNs failed to generate the target sequence and LSTMs improved with network size.The smallest CW-RNN was roughly on par with the second-biggest LSTM, and all but the smallest CW-RNN had significantly less variance than the other methods.
- Spoken word classification: The classification task used TIMIT spoken-word audio, where suffix similarity required learning long-term dependencies to disambiguate words.The dataset contained 175 sequences across 25 word classes, with MFCC and energy features as inputs.
- Results summary: Table 3 reports mean error and standard deviation over 100 runs for the largest models on both tasks.CW-RNN was 5.7× better than LSTM on sequence generation and more than 2× better on spoken-word classification.
- Spoken word classification: CW-RNNs beat both RNN and LSTM networks by a considerable margin of 8-20% on average, irrespective of parameter count.RNNs performed worst, while LSTMs improved substantially with more parameters.
5. Discussion
The discussion attributes CW-RNN’s performance to running hidden-unit subsets at different speeds, allowing the model to learn dynamic time-scales in complex signals. It also identifies open design choices and scope boundaries for future evaluation.
- Interpretation: Running subsets of neurons at different speeds allows an RNN to efficiently learn the different dynamic time-scales inherent in complex signals.This is the paper’s stated interpretation of the experimental results.
- Design choices: Alternative module-period schedules, including linear, Fibonacci, logarithmic, and random series, were not evaluated because exponential timing worked well in preliminary experiments.Learning the periods would require a differentiable modulo trigger or evolutionary training.
- Design choices: The lowest clock period can exceed 1, preventing output changes at every time step and potentially providing low-pass filtering for noisy data.The paper presents this as an optional design choice.
- Design choices: Module sizes and clocks could be adjusted using expected input information, such as frequency analysis of the data spectrum.The paper does not report experiments testing this adjustment.
- Scope boundary: Giving each weight its own clock was hard to train and produced poor results in initial experiments not included in the paper.Grouping neurons into modules is described as an intermediate design between a single clock and per-weight clocks.
- Speech interpretation: On speech classification, CW-RNN modeled and recognized complete words directly rather than first translating signal coefficients into phonemes.The authors state that multiple-speed modules benefit this direct approach.
- Future work: Future work includes analyzing internal dynamics, testing other problem classes such as reinforcement learning, and comparing against more connectionist sequence models.These plans indicate that the reported evidence remains limited to the preliminary experiments presented.
Appendix
The appendix shows that CW-RNN reduces recurrent parameters and per-step computation relative to a standard RNN with the same number of neurons. With exponentially scaled module periods, recurrent operations are faster by at least g/2, typically yielding a 2–5-fold speed-up.
- Architecture: CW-RNN partitions n = kg neurons into g modules of size k, with modules evaluated at prescribed temporal periods.Each module is evaluated only every T_i-th time step.
- Parameter count: CW-RNN uses roughly half as many recurrent-matrix parameters as an RNN with n neurons.The comparison baseline is the standard RNN recurrent matrix with n^2 parameters.
- Computation: For exponentially scaled periods, T_i = 2^i, the appendix derives an upper bound on recurrent operations per time step.The bound follows from evaluating each module only at its prescribed period.
- Speed: Recurrent operations are faster than in an equal-size RNN by at least g/2, typically producing a speed-up between 2 and 5.The stated typical range applies to common CW-RNN sizes.
- Speed: The appendix also derives expressions for input-weight evaluations and the overall CW-RNN speed-up relative to an RNN.These quantities are introduced after the recurrent-operation bound.
- Qualification: The reported parameter-count comparison is explicitly described as a conservative lower bound.This qualification limits how strongly the parameter reduction should be interpreted.