Source-linked AI summary
Recurrent Orthogonal Networks and Long-Memory Tasks
Mikael Henaff, Arthur Szlam, Yann LeCun
TL;DR
RNNs are difficult to train to use information from many timesteps earlier, motivating research into long-term dependencies. The paper constructs explicit solutions for two synthetic memory tasks and finds that identity initialization suits addition while random orthogonal initialization suits fixed-length copy. These constructions explain the initialization results and motivate pooling to choose between memory representations.
Problem
RNNs can be difficult to train to use information from inputs hundreds of timesteps earlier, despite their ability in principle to maintain hidden-state information over long sequences.
Source: §1 · p. 1, §2.1 · p. 2Method
The paper analyzes two synthetic long-memory tasks, constructs explicit RNN solutions, and tests initialization choices and l2 pooling.
Source: §1 · p. 2, §2.3 · p. 3Results
The fixed-length copy task has an explicit solution using a transition matrix that is a T + S root of identity, while identity initialization supports addition; reversing these initializations leads to poor results.
Source: §5 · p. 8Takeaways & Limitations
The constructions explain the success of orthogonal initializations and unitary constraints, while l2 pooling lets a model choose between identity-like and random-orthogonal-like representations.
Source: §1 · p. 2, §2.3 · p. 3For variable-length copy, the authors do not know a bounded-in-T explicit LT-RNN or sRNN solution.
Source: §3.1.3 · p. 5
Abstract
from arXiv · showhide
Although RNNs have been shown to be powerful tools for processing sequential data, finding architectures or optimization strategies that allow them to model very long term dependencies is still an active area of research. In this work, we carefully analyze two synthetic datasets originally outlined in (Hochreiter and Schmidhuber, 1997) which are used to evaluate the ability of RNNs to store information over many time steps. We explicitly construct RNN solutions to these problems, and using these constructions, illuminate both the problems themselves and the way in which RNNs store different types of information in their hidden states. These constructions furthermore explain the success of recent methods that specify unitary initializations or constraints on the transition matrices.
1. Introduction
RNNs can in principle retain information over long sequences, but vanishing gradients make them difficult to train for long-term dependencies. The paper constructs explicit solutions to synthetic long-memory tasks and uses them to explain the role of transition-matrix initialization.
1. Introduction
- Vanishing gradients remain difficult to address: when a transition matrix’s spectral norm is below 1, gradients vanish exponentially, while LSTMs use gates to control information flow.Other approaches include contextual memory, activation stabilization, and more sophisticated optimization; recent methods use orthogonal or unitary transition matrices.Source: §1 · p. 1
- Synthetic long-memory tasks require storing information for hundreds of timesteps, and prior methods solved the addition and copy tasks to varying extents.Reported results range from solving addition at T = 200, 300, or 400, to beating chance in 8/9 cases at T = 400; one method completely solved addition through T = 400 and partially at T = 750, and solved copy through T = 500.Source: §1 · p. 2
- The paper constructs explicit RNN solutions and finds that random orthogonal initialization suits sequence memorization, identity initialization suits addition, and l2 pooling lets a model choose between memory representations.Experiments show that correct initialization is critical on these tasks.Source: §1 · p. 2
- The analysis complements earlier work on orthogonal and generic linear RNNs by addressing storage of discrete input sequences.The cited earlier work studies storage of scalar sequences in hidden states.Source: §1 · p. 2
2. Architectures
The section introduces recurrent architectures and modifications used for long-memory tasks. It describes standard and linear-transition RNNs, LSTMs, and an l2-pooling variant that combines pooled and unpooled representations.
2. Architectures
- The section establishes notation for sequential inputs and outputs before describing the architectures used for the long-memory problems.The input and output sequences are denoted x0, x1, ..., xt, ... and y0, y1, ..., yt, ... .Source: §2 · p. 2
2.1. sRNN
- An sRNN updates a hidden state from sequence inputs and uses it to produce outputs, but can be difficult to train to use inputs from hundreds of timesteps earlier.The paper uses LT-RNNs, which place nonlinearity between input and hidden state rather than hidden state and output; an sRNN can approximate an LT-RNN with appropriately scaled weights and biases, but not vice versa.Source: §2.1 · p. 2
2.2. LSTM
- The LSTM architecture uses memory cells with update, input, forget, and output gates, and computes cell and hidden states from the input, previous state, and output gate.The section also notes peephole connections, which let cell-state information flow to gates and were designed to measure or generate precise time intervals.Source: §2.2 · pp. 2–3
2.3. LT-RNN with l2 pooling
- The l2-pooling LT-RNN feeds pooled and unpooled hidden layers to the decoder, allowing it to choose an identity-like or random-orthogonal-like representation.The model uses a fixed pool size k; P(h) maps a kd-dimensional hidden vector to a d-dimensional vector.Source: §2.3 · p. 3
3. Tasks
The section defines long-memory copying and addition tasks and gives explicit recurrent mechanisms for them. It contrasts the transition structures suited to each task and examines fixed- and variable-length copying.
3. Tasks
- The tasks are designed to test storage over hundreds of timesteps and to be pathologically difficult for standard sRNNs.The section identifies these tasks as long-timescale dependencies.Source: §3 · p. 3
3.1. Copying Problem
- The copying task requires recalling S categorical symbols across T steps, then outputting them in their original order after a delimiter.The required output begins with T+S blank symbols; the objective is average cross-entropy across time steps.Source: §3.1 · p. 3
3.1.1. A SOLUTION MECHANISM
- Copying uses a periodic rotation clock to hide encoded symbols, then surface them in input order after the delimiter; the construction’s success depends on dimension being large enough.The authors argue that with sufficiently large d, symbol scores separate with high probability, and nearest-neighbor interference is exponentially unlikely for fixed ϵ and K.Source: §3.1.1 · pp. 3–4
3.1.2. SOLUTION MECHANISM EXPERIMENTS
- At T=500 and d=128, Figure 1 reports success percentage over 500 trials; the experiments examine how success varies with dictionary size K and sequence length S.The passage reports no specific success percentages in the supplied text.Source: Figure 1 · p. 4, §3.1.2 · p. 5
3.1.3. VARIABLE LENGTH COPY PROBLEM
- The fixed-delimiter copy mechanism does not directly address a randomly located output signal; the authors know no bounded-in-T LT-RNN or sRNN solution for that variant.The passage notes a multiplicative-RNN alternative with extra hidden variables to track the power of V.Source: §3.1.3 · p. 5
3.2. Adding Problem
- The adding task presents two marked numbers in a sequence and requires their sum as output.Each input has a value coordinate and a marker coordinate.Source: §3.2 · p. 5
3.2.1. A SOLUTION MECHANISM
- A one-dimensional ReLU LT-RNN solves addition by leaving the hidden state unchanged on unmarked inputs and adding xj[2] when xj[2]=1.The passage specifies U=[11], b=−1, W=1, and V=1.Source: §3.2.1 · p. 5
3.3. Comparison between the tasks
- The comparison assigns identity-like transitions to addition and diverse unit-circle eigenvalues to copying; l2 pooling offers access to identity-like or clock-like representations.The copy-task clock synchronizes after S+T steps, while its blocks should look random between synchronizations.Source: §3.3 · p. 5
4. Experiments
The experiments compare initialization and pooling on fixed-length copy and addition tasks, then test whether those results extend to variable-length copying. Orthogonal and identity initialization show task-dependent performance, pooling broadens performance across tasks, and the variable-length test favors the LSTM.
4.1. Impact of Initialization
- LT-ORNN solves fixed-length copy almost immediately, while LT-IRNN does not; on addition, LT-IRNN solves efficiently, whereas LT-ORNN may take much longer or fail.The copy-task models used 80 hidden units and the addition-task models used 128; for T = 750, only one LT-ORNN and one LT-IRNN were trained due to time constraints.Source: Figure 2 · p. 6, Figure 3 · p. 6, §4.1 · pp. 6–7
4.2. Pooling Experiments
- Pooling lets LT-ORNN solve copy at both tested timescales and approximately solve addition, though it converges more slowly than LT-IRNN.The activation analysis suggests that rotations store information in their radius and pooling discards phase to recover it as a delta-like signal.Source: Figure 4 · p. 7, §4.2 · pp. 7–8, Figure 6 · p. 8
4.3. Variable Length Copy Task
- On variable-length copy at T = 100, none of the tested LT-RNNs beat chance, while the LSTM solves the task slowly.The authors note that a synthetic benchmark's construction can favor a model in ways that fail to generalize to other tasks.Source: Figure 5 · p. 8, §4.3 · p. 8
5. Conclusion
The conclusion gives explicit RNN solutions for two synthetic long-memory tasks and links their dynamics to task-appropriate transition-matrix initialization. It also cautions that success on one synthetic benchmark may not generalize to another task.
5. Conclusion
- The fixed-length T copy problem is solvable with a transition matrix that is a (T + S)th root of identity I and has eigenvalues distributed around the unit circle.Random orthogonal matrices almost satisfy this description.Source: §5 · p. 8
- The addition problem can be solved with identity I as the transition matrix.Source: §5 · p. 8
- Identity initialization makes a linear-transition RNN easy to optimize for addition, while random orthogonal initialization does so for copy; reversing them leads to poor results.An l2 pooling layer can mitigate the optimization difficulty of transitioning between oscillatory and steady dynamics by letting the model choose between regimes.Source: §5 · p. 8
- Success on the variable-length copy task illustrates that performance on one synthetic benchmark does not necessarily generalize across tasks.The authors recommend evaluating novel architectures on broad benchmarks and natural data.Source: §5 · p. 8