Source-linked AI summary
Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks
Ralf C. Staudemeyer, Eric Rothstein Morris
TL;DR
The paper addresses the difficulty of understanding how LSTM-RNNs evolved and why they work, amid differing notation and errors across earlier publications. It provides a detailed tutorial, unified notation, and explanatory figures centered on the vanishing-error problem and LSTM’s constant error flow. The paper concludes that LSTM can handle time lags exceeding 1,000 steps and describes extensions for self-resets and precise timing.
Problem
Earlier LSTM publications used differing notation and accumulated errors or inconsistencies, making the networks’ evolution and derivations difficult to follow.
Method
The paper presents a detailed tutorial of LSTM and its extensions, revising notation and documentation while explaining key mechanisms and early publications.
Results
LSTM addresses RNN vanishing errors through constant error flow in special memory cells, enabling time-lag handling beyond 1,000 steps.
Takeaways & Limitations
The paper presents constant error flow as the central LSTM mechanism and introduces extensions that enable self-resets and precise timing.
Takeaways & Limitations
LSTM requires a fixed network topology, so its memory is ultimately limited because the number of memory blocks does not change dynamically.
Abstract
from arXiv · showhide
Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) are one of the most powerful dynamic classifiers publicly known. The network itself and the related learning algorithms are reasonably well documented to get an idea how it works. This paper will shed more light into understanding how LSTM-RNNs evolved and why they work impressively well, focusing on the early, ground-breaking publications. We significantly improved documentation and fixed a number of errors and inconsistencies that accumulated in previous publications. To support understanding we as well revised and unified the notation used.
1 Introduction
The article introduces LSTM-RNNs by tracing their evolution from feed-forward networks and RNNs, while unifying notation and adding detailed explanations to support understanding. It focuses on the vanishing or exploding signals that limit ordinary RNNs and motivate LSTM.
- The paper develops a tutorial-like introduction for readers seeking to understand LSTM-RNNs and their evolution since the early 1990s.
- Feed-forward neural networks support static mappings, whereas time-dependent prediction requires dynamic classifiers.
- RNNs add recurrent connections to incorporate signals from previous timesteps, but their usable temporal context is limited by vanishing or exploding signals.
- LSTM-RNNs address the temporal limitations of ordinary RNNs and are presented through their evolution from simpler neural-network architectures.
- The authors unify notation and provide descriptive figures because early LSTM publications used differing notations that made development and derivations difficult to follow.
- The article claims unusually detailed coverage of LSTM and its extensions.
2 Notation
This section establishes notation for units, connections, signals, states, errors, timing, and network subsets used throughout the article. The notation distinguishes inputs, outputs, predecessors, successors, and learning quantities.
- η denotes the network learning rate, τ a time unit, and t′ and t the initial and final times of an epoch.
- The notation defines network units as elements of N, with generic units u, v, l, and k belonging to that set.
- Input and output units are collected in I and O, while non-input units are collected in U.
- The unit output or activation is y_u, while e_u and ϑ_u denote its error and error signal.
- Pre(u) and Suc(u) denote the predecessors and successors of unit u, respectively.
- W[v,u] denotes the weight from v to u, and X[v,u] denotes the input to u coming from v.
- The notation assigns z_u, b_u, s_u, and f_u to a unit’s weighted input, bias, state, and squashing function.
- p_k denotes the output sensitivity of unit k with respect to weight W[u,v].
3 Perceptron and Delta Learning Rule
Perceptrons compute thresholded weighted sums and can represent only linearly separable functions, while delta learning and sigmoid units extend training and representational capabilities.
- Perceptron: A perceptron sums weighted real-valued inputs with a bias and fires when its state exceeds the threshold.Its output is Boolean, with the deactivated value usually −1 and a typical threshold of 0.
- Linear Separability: Single perceptrons represent functions such as AND, OR, NAND, and NOR but are limited to linearly separable decision surfaces.OR is linearly separable, whereas XOR is not; the trained weights represent the separating line.
- Perceptron Training: Supervised perceptron training compares predictions with target values and modifies weights after misclassification.Finite-time convergence is guaranteed for linearly separable training examples, but not for non-separable data.
- Delta Learning Rule: The perceptron learning rule converges only when training data is linearly separable and the learning rate is sufficiently small.Its initial weight vector contains random values, and the rule fails on non-linearly separable examples.
- Delta Learning Rule: The delta learning rule handles both linearly separable and non-separable examples by modifying weights through gradient descent toward lower error.It calculates errors between computed outputs and training targets before updating the weights.
- Sigmoid Threshold Unit: Sigmoid threshold units replace Boolean outputs with continuous values between 0 and 1 and can represent non-linear functions.The sigmoid squashes a large input domain into a small output range, approaching zero for low input and one for high input.
4 Feed-Forward Neural Networks and Backpropagation
Feed-forward networks organize neurons into loop-free layers and use differentiable activations with backpropagation to learn weights by reducing output error.
- Network Structure: Feed-forward neural networks organize input, hidden, and output neurons in layers, with hidden neurons not directly connected to the environment.They are loop-free and fully connected between successive layers.
- Network Structure: Single-layer perceptron networks directly connect input neurons to output neurons, applying weights to the input-output connections.Each perceptron fires 1 above its threshold and otherwise takes the deactivated value, usually −1.
- Network Structure: Multilayer feed-forward networks connect input and output layers through hidden layers and can express non-linear decision surfaces using sigmoid threshold functions.Given enough hidden units, these networks can closely approximate any function.
- Backpropagation: Backpropagation uses gradient descent in small iterative steps from the output layer toward the input layer and requires differentiable activation functions.Weights are commonly initialized to small normalized random values before training samples are processed.
- Forward Propagation: Forward propagation passes inputs through the layers until output units produce the observable network output.The output of each output unit corresponds to a component of the network output vector.
- Backpropagation: Backpropagation compares each output with its desired target, computes output and overall network errors, and updates weights and biases to reduce error.Hidden-unit errors are obtained by propagating error signals backward through successor units until weight changes satisfy a terminating condition.
5 Recurrent Neural Networks
Recurrent neural networks extend neural classifiers with internal state and feedback connections, allowing earlier events to influence current processing and enabling time-series memory.
- RNNs are dynamic systems with an internal state at each classification time step.
- Feedback connections propagate data from earlier events to current processing steps, allowing RNNs to build memory of time-series events.
- Elman networks save hidden-layer outputs in context cells that feed back to the corresponding hidden neurons.
- RNN architectures range from partially recurrent networks with hidden-layer self-feedback to fully recurrent networks with self-feedback connections.
- RNN training differs from feed-forward training because information must be propagated through recurrent connections between time steps.
6 Training Recurrent Neural Networks
Recurrent networks are trained by handling temporal dependencies through either time-unfolded backpropagation or forward-propagated sensitivity information. BPTT aggregates updates across an epoch, whereas RTRL learns incrementally during online input presentation.
- The original LSTM formulation used a combination of BPTT and RTRL.
- Backpropagation Through Time: BPTT unfolds an RNN into a feed-forward network with one layer per time step and shared weights.The unfolded network has identical behavior to the recurrent network over a finite time period.
- Backpropagation Through Time: BPTT backpropagates error through the unfolded sequence and sums corresponding weight deltas across all time steps.
- Backpropagation Through Time: BPTT computes error signals for units across an epoch whose cost is the summed error over the sequence.
- Real-Time Recurrent Learning: RTRL forward-propagates gradient information while the input stream is presented, eliminating a dedicated training interval.The algorithm uses non-local sensitivity information and has significant computational cost per update cycle.
- Backpropagation Through Time: The gradient of total error is accumulated over previous and current time steps before determining the overall recurrent weight change.
- Real-Time Recurrent Learning: RTRL recursively updates sensitivities and combines them with each time step's error vector to calculate weight changes incrementally.
7 Solving the Vanishing Error Problem
Standard RNNs struggle with long-term dependencies because backpropagated errors can exponentially grow or shrink across time. The vanishing-error case prevents learning within an acceptable period, motivating gradient-based solutions such as LSTM.
- 5–10 time steps is the approximate maximum dependency span of standard RNNs because back-propagated error signals grow or shrink over time.
- Growing error products can cause exploding errors, oscillating weights, and unstable learning.
- Shrinking error products cause the error to vanish and prevent learning within an acceptable time period.
- The analysis links local error vanishing to global error vanishing.
- The vanishing-error problem motivated proposals including the gradient-based long short-term memory method.
8 Long Short-Term Neural Networks
LSTM addresses vanishing gradients by preserving constant error flow in memory cells and regulating access with multiplicative gates. Its memory block extends the constant-error mechanism to manage incoming signals and memory readout.
- LSTM can learn minimal time lags of more than 1,000 discrete time steps using constant error carousels and multiplicative gates.The gates learn when to grant access to the memory cells.
- Constant Error Carousel: A constant error carousel requires a linear activation and constant cell activation over time.
- Constant Error Carousel: The CEC uses the identity function and recurrent weight W[u,u] = 1.0 to preserve error flow.This preservation is the central feature of LSTM and supports extended short-term memory storage.
- Additional weighted inputs and outputs create conflicting update signals because the same connections can store, ignore, retrieve, or suppress information.
- LSTM extends the CEC with input and output gates, forming a memory block that regulates network access to the cell.
9 Training LSTM-RNNs - the Hybrid Learning Approach
LSTM combines gated memory cells with a hybrid BPTT/RTRL training approach. Its fixed self-connection preserves state, while forget gates address unbounded state growth; per-step computation remains O(1), but topology and memory capacity are fixed.
- Memory block: Each memory block uses an input gate and output gate to regulate cell-state updates and cell outputs.The cell state combines squashed weighted input with the previous state, while the output gate regulates the squashed cell-state output.
- Gated state updates: Gate activations are squashed into [0, 1], so the memory-cell input passes only when the input-gate signal is sufficiently close to 1.The cell-input squashing function scales values to [−2, 2], while the output squashing function has range [−1, 1].
- Limitations: Fixed-weight self-connections can cause cell states to grow linearly, eliminating memorisation and making the cell behave like an ordinary RNN neuron.Manually resetting state can limit growth, but is impractical for continuous input streams without clear sequence boundaries.
- Forget gates: A forget gate replaces the fixed self-connection weight of 1.0, allowing LSTM to learn when to reset stored cell information.The forget-gate bias is fixed to 1 following a cited recommendation to improve performance.
- Hybrid learning approach: LSTM separates units between BPTT and RTRL: output, hidden, and output-gate units use BPTT, while input gates, forget gates, and cells use RTRL.The original formulation uses RTRL for cell-related derivatives that must be computed at every step, including steps without target values.
- Complexity and scope: LSTM’s computational complexity per step and weight is O(1), but its fixed topology ultimately limits network memory.The paper notes that homogeneous network scaling is unlikely to overcome this limitation and that modularisation is not generally clear.
10 Problem specific topologies
LSTM topologies are extended for task-specific sequence directions, segmentation, depth, and multidimensional inputs. These variants alter information flow, while GRU offers a gated alternative without a memory cell.
- Bidirectional LSTM: Bidirectional LSTM processes inputs forwards and backwards with separate LSTM networks connected to the same output layer.Bidirectional training removes the original one-step truncation and permits full error-gradient calculation using standard BPTT.
- CTC: Connectionist Temporal Classification enables LSTM-RNNs to handle input data that is not segmented into sequences.This is useful when correct segmentation, such as separating letters in handwriting, is difficult.
- Grid LSTM: Grid LSTM generalises LSTM’s input-selection ability to deep networks by arranging LSTM cells along and between dimensions.It adds cells along the depth dimension and modulates interactions among layers to avoid instability associated with multidimensional LSTM.
- Stacked LSTM: Stacked LSTM increases capacity by placing LSTM layers on top of one another, using hidden signals from the previous transform as later-layer inputs.The first network uses the original input signals, while subsequent networks replace them with preceding hidden signals.
- Multidimensional LSTM: Multidimensional LSTM processes grid-structured inputs with N recurrent connections, receiving N hidden and memory vectors and producing one of each.Its memory calculation combines forget signals with memory vectors using the Hadamard product.
- GRU: GRU provides a gated alternative to LSTM without a memory cell, using reset and update gates to control recurrent activation.The update gate emulates the LSTM forget and input gates through complementary factors in the activation equation.
11 Applications of LSTM-RNN
The paper surveys LSTM-RNN applications across speech, handwriting, translation, image processing, real-world, and computational tasks, documenting strong results and continued architectural development.
- Applications overview: The surveyed applications show LSTM-RNNs addressing a large variety of cognitive learning tasks, including speech, handwriting, translation, emotion recognition, and text generation.The paper identifies speech and handwriting recognition and machine translation as predominant literature applications.
- Speech recognition: LSTM-RNN architectures achieved strong results across speech recognition, including BLSTM-CTC systems that outperformed HMMs and later deep variants with outstanding results.Large-vocabulary speech recognition also reported best results with an LSTM/HMM hybrid architecture.
- Handwriting recognition: BLSTM-CTC enabled handwriting systems to transcribe raw online data directly and later outperformed Hidden-Markov-based recognition systems.A real-world system combining BLSTM-CTC with a probabilistic language model achieved an error rate comparable to a human on the task.
- Machine translation: Sequence-to-sequence and attention-based LSTM architectures improved machine translation, including handling rare words and long sentences.The attention mechanism was implemented in the decoder to address translation issues involving long sentences.
- Image processing: LSTM variants were applied to image and document tasks, outperforming HMMs and SVMs in handwritten-content classification and improving ImageNet results.Sequence-to-sequence models also generated English image descriptions, while hierarchical visual features supported activity recognition and image/video description.
- Other learning tasks: Applications extended to protein structure prediction, music generation, network security, program evaluation, and combinatorial optimisation.Sequence-to-sequence frameworks were used both to evaluate short computer programs and to learn solutions to combinatorial optimisation problems.
12 Conclusions
The conclusion presents LSTM as a response to RNN vanishing errors, using constant error flow to handle long time lags and extensions for self-resets and precise timing.
- Core contribution: LSTM addresses the vanishing error problem by introducing constant error flow through the internal states of special memory cells.The paper identifies vanishing error as a serious shortcoming of RNNs.
- Long-term dependencies: LSTM can bridge time intervals in excess of 1,000 time steps, enabling it to tackle long time-lag problems.This capability follows the paper’s description of constant error flow through memory-cell states.
- Extensions: Two LSTM extensions enable learning self-resets and precise timing, with self-resets freeing memory from irrelevant information.The conclusion presents these extensions as additions to the core LSTM mechanism.