PaperlayerView exact revision on arXiv

Source-linked AI summary

Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

Alex Sherstinsky

arXiv:1808.03314v10cs.LGstat.ML

TL;DR

Existing RNN/LSTM resources often omit training equations and leave RNN unrolling unjustified, while lacking a single clear, complete primer. This paper derives RNN fundamentals from differential equations, formally justifies unrolling, and logically constructs Vanilla LSTM, yielding a principled, comprehensive tutorial for understanding and implementing these systems.

  • Problem

    Existing resources lack a single clear, self-contained treatment of RNN/LSTM fundamentals, often omitting training equations and justification for RNN unrolling.

  • Method

    The paper derives canonical RNN equations from differential equations, proves conditions for unrolling, and constructs Vanilla LSTM from the RNN through stability and training analyses.

  • Results

    The paper formally explains RNN unrolling and presents complete inference and training equations for Vanilla LSTM, including gated error-gradient behavior and CEC operation under specified parameters.

  • Takeaways & Limitations

    The tutorial provides a principled, intuitive reference for researchers and practitioners seeking to understand the reasoning behind and implement RNN and LSTM systems.

Abstract

from arXiv · show

Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles, the inference formulas for the LSTM network and its parent, RNN, are stated axiomatically, while the training formulas are omitted altogether. In addition, the technique of "unrolling" an RNN is routinely presented without justification throughout the literature. The goal of this paper is to explain the essential RNN and LSTM fundamentals in a single document. Drawing from concepts in signal processing, we formally derive the canonical RNN formulation from differential equations. We then propose and prove a precise statement, which yields the RNN unrolling technique. We also review the difficulties with training the standard RNN and address them by transforming the RNN into the "Vanilla LSTM" network through a series of logical arguments. We provide all equations pertaining to the LSTM system together with detailed descriptions of its constituent entities. Albeit unconventional, our choice of notation and the method for presenting the LSTM system emphasizes ease of understanding. As part of the analysis, we identify new opportunities to enrich the LSTM system and incorporate these extensions into the Vanilla LSTM network, producing the most general LSTM variant to date. The target reader has already been exposed to RNNs and LSTM networks through numerous available resources and is open to an alternative pedagogical approach. A Machine Learning practitioner seeking guidance for implementing our new augmented LSTM model in software for experimentation and research will find the insights and derivations in this tutorial valuable as well.

I. INTRODUCTION · II. THE ROOTS OF RNN

The paper presents a self-contained, intuitive, complete, and general tutorial intended to unify the fundamentals of RNNs and LSTMs. It derives the canonical RNN from differential equations, using the simpler RNN as the logical foundation for understanding LSTM architecture.

  • I. INTRODUCTION: The tutorial addresses gaps in existing resources by gathering the essential RNN and LSTM theory into one unifying reference.It is motivated by resources that leave basic questions unanswered and by the lack of a single self-contained primer on the Vanilla LSTM computational cell.
  • I. INTRODUCTION: The authors aim to explain RNN and LSTM fundamentals with consistent, meaningful notation that removes mystery and supports learning by students and practitioners.The document is explicitly framed as introductory text for future students, practitioners, and inquisitive researchers.
  • I. INTRODUCTION: The paper begins with RNNs because LSTMs are a type of RNN, and the simpler system provides intuition and a logical path toward the LSTM architecture.The canonical RNN equations are derived from differential equations and used as the starting model for the subsequent development.
  • I. INTRODUCTION: The treatment is designed to be intuitive, complete, and general by covering descriptive notation, inference and training equations, all system components, and the most inclusive Vanilla LSTM form.The stated requirements include both forward-pass and backward-pass equations and account for all components of the system.
  • II. THE ROOTS OF RNN: The canonical RNN formulation is derived from a general nonlinear first-order non-homogeneous ordinary differential equation describing a d-dimensional state signal.The derivation introduces a time-dependent state vector, an input signal vector, and a vector-valued function governing state evolution.
  • II. THE ROOTS OF RNN: The continuous-time model incorporates delayed state, readout, and external-input contributions, whose temporal structure supplies memory and captures causal or contextual information.The three delayed components are distinguished as analog state, warped readout, and external input terms, alongside a constant bias.
  • II. THE ROOTS OF RNN: Applying time discretization converts the delay differential system into a nonlinear first-order non-homogeneous difference equation over discrete-time sequences.The construction uses backward Euler discretization, sets the delay to one sampling step, and interprets the delayed readout as stored memory overwritten at each step.
  • II. THE ROOTS OF RNN: The resulting discrete system is transformed into the canonical Recurrent Neural Network form, whose stability is governed by the matrix ˆW ≈ Wr ≈ −A−1B.Under the special case B = ΛB, the stability analysis reduces to the diagonal matrix ˜W = −A−1ΛB.

III. RNN UNFOLDING/UNROLLING

RNN unrolling evaluates a recursively defined cell from specified initial conditions over a finite number of steps, approximating its inherently infinite-memory behavior with a finite impulse response. This approximation is justified under independent-subsequence and state-initialization conditions, but truncation can introduce discontinuities and fail on long-range dependencies.

  • Finite-step unrolling: Unrolling specifies the initial state and numerically evaluates the recursive RNN equations over a finite range of discrete steps.The resulting sequence of steps is generated from the RNN cell by repeated evaluation.
  • Infinite-memory behavior: Because the state recurrence repeatedly incorporates earlier states and inputs, each state s[n] contains contributions from all preceding indices, making the standard RNN an infinite impulse response system.The response to a single impulse at n = 0 remains defined for every positive n.
  • Theoretical justification: Proposition 1 justifies finite unrolling when a length-N target sequence can be partitioned into finite, mutually independent subsequences with suitable state initialization.Under these assumptions, the equations reduce to a standard RNN unrolled for Km steps on each segment.
  • Limitations of truncation: Truncation approximates the RNN by a finite impulse response system, but longer unrollings improve fidelity while reducing efficiency and can introduce artificial discontinuities.No IIR-to-FIR conversion is universally optimal, and short unrollings may fail when the target contains extremely long-range dependencies or dependent subsequences.

IV. RNN TRAINING DIFFICULTIES

Training a truncated unrolled RNN with BPTT propagates gradients backward through shared parameters and recurrent state dependencies. Over long-range dependencies, these gradients can vanish or explode, making standard Gradient Descent unreliable despite forward-system stability.

  • Gradient instability: Consequently, training standard RNNs on long windows with Gradient Descent is hampered by vanishing or exploding gradients even when the unrolled system is stable by design.Regulating the backpropagated gradient signal remains challenging, leaving no reliable parameter-update mechanism.
  • BPTT: BPTT trains truncated unrolled RNNs by repeatedly applying the chain rule after converting the recurrent computation graph into a directed acyclic graph.This enables sequence-specific backpropagation through the unrolled steps.
  • Gradient dependencies: Because the state at index n influences later states through the recurrent connection, total gradients must account for both direct readout and future-state dependencies.Ignoring the recurrent dependency would omit an important component of the gradient with respect to the state signal.
  • Parameter updates: The same parameter set is shared across all unrolled steps, so the parameter gradient aggregates objective-function contributions from the entire sequence.This aggregated derivative is used during optimization of the RNN parameters.
  • Gradient instability: Long-range dependencies make gradient propagation numerically difficult: recurrent Jacobian factors can shrink toward zero or grow exponentially when stability conditions are violated.Growth may drive the state toward warping-function saturation or cause overflow, while shrinking produces vanishing gradients.

V. FROM RNN TO VANILLA LSTM NETWORK

The Vanilla LSTM transforms the standard RNN by adding learned gates that separately regulate state retention and update injection, addressing vanishing gradients while preserving adaptive sequence processing. Its constant-error mode can propagate gradients unattenuated, although gradients may still vanish when state-retention gates remain below one.

  • Motivation: LSTM gates were introduced to address vanishing gradients by learning nonlinear, data-dependent controls over the state signal’s gradient.The gradient with respect to the state signal is directly proportional to parameter updates during Gradient Descent.
  • Gating controls: The Vanilla LSTM updates its state by separately weighting the previous-state contribution and the current update information.The update is s[n] = gcs[n] ⊙ Fs(s[n −1]) + gcu[n] ⊙ Fu(r[n −1],x[n]).
  • Vanilla LSTM cell: The complete Vanilla LSTM cell combines gating controls with signal containment so its state aggregates historical and novel update information.The previous-state contribution remains fractional, supporting overall system stability.
  • Gradient propagation: When gcs[n] = 1 and gcu[n] = gcr[n] = 0, the cell forms a Constant Error Carousel that recirculates the error gradient without attenuation.Under these settings, s[n] = s[n −1] and ψ[n] = ψ[n + 1] throughout the segment.
  • Limitations: Vanilla LSTM gradients can still vanish over finite steps when the learned state-retention gate satisfies ∥gcs[n]∥ < 1.The network nevertheless accommodates Gradient Descent better than the standard RNN because of its gates.

VI. THE VANILLA LSTM NETWORK MECHANISM IN DETAIL · A. Overview

The Vanilla LSTM cell processes an input sequence step by step, maintaining an internal state and producing an externally accessible signal. Its mechanism separates candidate-data preparation from data control, using three update, state, and readout stages.

  • A. Overview: At step n of an unrolled K-step sequence, the LSTM cell accepts input x[n] and computes observable output v[n].The cell maintains its internal state in s[n].
  • A. Overview: The cell’s internal state s[n] is normally inaccessible to entities outside the cell.This distinguishes the maintained internal state from the externally accessible signal.
  • A. Overview: LSTM operations organize around two cooperating objectives: preparing data and controlling how data propagates.The data and control components perform distinct but coordinated roles.
  • A. Overview: Data components generate candidate-data signals ranging from −1 to 1.These signals represent the data prepared for possible propagation.
  • A. Overview: Control components generate throttle signals ranging from 0 to 1.The throttle determines the fractional amount of candidate data allowed to propagate.
  • A. Overview: Multiplying candidate-data and control signals apportions the candidate data that propagates through the cell.The control signal therefore regulates candidate-data transmission.
  • A. Overview: The Vanilla LSTM cell contains three candidate-data/control stages: update, state, and readout.These stages are depicted in Figure 7.
  • A. Overview: In advanced RNN and LSTM configurations such as Attention Networks, the state signal can become externally observable and contribute to the objective function.This is an exception to the usual inaccessibility of the internal state.

B. Notation

This section defines the notation for the Vanilla LSTM cell, including step indexing, segment length, signal dimensions, gates, state-related signals, and training quantities.

  • Indexing and dimensions: n indexes steps from 0 to K −1, while K denotes the number of steps in the unrolled segment.The segment may also be called a subsequence.
  • Warping functions: Gc is a monotonic, bipolarly saturating gate function, whereas Gd is a monotonic, negative-symmetric, bipolarly saturating function for bounding data.
  • Indexing and dimensions: dx and ds specify the input-signal and state-signal dimensionalities, respectively.
  • Cell signals: The cell notation distinguishes observable value, accumulation, update-candidate, and gate-output signals, represented by v, a, u, and g.v and a belong to R^ds; u and g also belong to R^ds.
  • Training and operations: E denotes the objective cost minimized during training, while inner, outer, and matrix-vector products yield scalars, matrices, and vectors, respectively.

C. Control/Throttling (“Gate”) Nodes

The Vanilla LSTM cell uses three gate types that regulate how candidate and prior-state signals form the current state and how the readout becomes externally observable.

  • Control/Throttling (“Gate”) Nodes: The update gate controls the fractional amount of the update candidate signal contributing to the cell’s present state at index n.
  • Control/Throttling (“Gate”) Nodes: The state gate controls the fractional amount of the adjacent lower-indexed state signal at n −1 contributing to the present state at n.
  • Control/Throttling (“Gate”) Nodes: The readout gate controls the fractional amount of the readout candidate signal released as the cell’s externally accessible observable signal at index n.

D. Data Set Standardization … 3) Parameters of the accumulation node,⃗acr[n], of the gate that controls the fractional amount of the readout candidate signal, r[n], used to release as the externally-accessible (observable) value signal of the cell at the present step with the index, n:

The paper standardizes inputs before processing, selects bounded nonlinearities for control and data, and organizes the Vanilla LSTM into fifteen parameter entities grouped by update, state, and readout gates.

  • D. Data Set Standardization: Input samples are standardized so each network input element has training-set mean 0 and standard deviation 1.The external data set x0[n] is transformed into standardized inputs x[n] before LSTM or RNN operation.
  • D. Data Set Standardization: The standardization transformations use the training-set sample mean, auto-covariance matrix, and sample count, and also apply training statistics to test and validation sets.N denotes the number of training samples, µ the sample mean, and V the sample auto-covariance matrix.
  • E. Warping (Activation) Functions: The logistic sigmoid is chosen for control because it is monotonic, continuous, differentiable, and bounded between 0 and 1.Its control output is represented as Gc(z) ≡ σ(z) ≡ 1/(1 + e^−z).
  • E. Warping (Activation) Functions: The hyperbolic tangent is used for data bounding because it is monotonic, negative-symmetric, and saturates at −1 and 1.This squashing behavior supports both negative and positive standardized inputs while keeping Gd(z) bounded.
  • F. Vanilla LSTM Cell Model Parameters: The Vanilla LSTM cell model uses fifteen parameter entities with specified dimensions and designations.These entities are organized around accumulation nodes controlling update, state, and readout behavior.
  • 1) Parameters of the accumulation node,⃗acu[n], of the gate that controls the fractional amount of the update candidate signal, u[n], used to comprise the state signal of the cell at the present step with the index, n:: The update-control accumulation node receives input, previous-state, and previous-observable-value weights plus a bias vector.The corresponding matrices are Wxcu ∈ R^ds×dx, Wscu ∈ R^ds×ds, and Wvcu ∈ R^ds×ds, with b⃗cu ∈ R^ds.
  • 2) Parameters of the accumulation node,⃗acs[n], of the gate that controls the fractional amount of the state signal of the cell, s[n −1], at the adjacent lower-indexed step, n −1, used to comprise the state signal of the cell at the present step with the index, n:: The state-control accumulation node likewise has input, previous-state, and previous-observable-value weight matrices together with a bias vector.These parameters connect x⃗[n], s⃗[n − 1], and v⃗[n − 1] to acs[n].
  • 3) Parameters of the accumulation node,⃗acr[n], of the gate that controls the fractional amount of the readout candidate signal, r[n], used to release as the externally-accessible (observable) value signal of the cell at the present step with the index, n:: The readout-control accumulation node uses input, current-state, and previous-observable-value weight matrices plus a bias vector.These parameters connect x⃗[n], s⃗[n], and v⃗[n −1] to acr[n] to control release of the readout candidate r[n].

4) Parameters of the accumulation node,⃗adu[n], for the data warping function that produces the update candidate signal, u[n], of the cell at the present step with the index, n: … I. Vanilla LSTM System Derivatives (“Backward Pass”)

The paper specifies the Vanilla LSTM’s data-update parameters, collects all learnable quantities into Θ, and defines the cell’s forward and backward computations. Its BPTT training derives backward-moving gradients, parameter derivatives, and sequence-wide gradient aggregation for optimization.

  • 4) Parameters of the accumulation node: The data-update accumulation node uses Wxdu to connect x[n], Wvdu to connect v[n −1], and bdu as its bias vector.Wxdu ∈Rds×dx, Wvdu∈Rds×ds, and bdu ∈Rds.
  • 5) All model parameters, which must be learned, combined (for notational convenience): •: All Vanilla LSTM parameters are concatenated into the single model-parameter collection Θ for notational convenience.The collection includes the control, state, reset, and data-update weights and biases.
  • G. Summary of the main entities (generalized): The generalized glossary organizes the model’s main entities without subscripts or indices, while reserving ψ, χ, α, ρ, and γ for definition in Section VI-I.This provides a notation-independent summary of the system entities.
  • H. Vanilla LSTM System Equations (“Forward Pass”): Each unrolled cell step depends on the preceding step’s characterized quantities, and Equations 126–136 fully define the Vanilla LSTM cell.Positive-direction unrolling uses n −1; opposite-direction and bidirectional variants reverse or combine the evaluation directions.
  • I. Vanilla LSTM System Derivatives (“Backward Pass”): BPTT first computes backward-moving gradient sequences χ[n] and ψ[n], whose values at n depend on the corresponding quantities at n + 1.χ[n] is the total derivative with respect to v[n], while ψ[n] is the total derivative with respect to s[n].
  • I. Vanilla LSTM System Derivatives (“Backward Pass”): The backward pass applies the chain rule to obtain accumulation and model-parameter derivatives, with accumulation derivatives clipped between −1 and 1 to prevent numerical problems.The parameter derivatives are computed at each step from the Vanilla LSTM cell equations.
  • I. Vanilla LSTM System Derivatives (“Backward Pass”): When unrolled for K steps, shared parameters Θ receive gradient contributions from every step, which are then used for Gradient Descent after summing gradients across batch segments.Typical batch sizes range between 16 and 128 segments.

J. Error Gradient Sequences in Vanilla LSTM System

The section analyzes how Vanilla LSTM gradient sequences preserve or lose error signals across long time ranges, emphasizing conditions that mitigate vanishing gradients. It identifies Constant Error Carousel recirculation as the characteristic mechanism for sustaining gradients when the state signal saturates at one.

  • Gradient-flow requirements: Vanilla LSTM training requires intermediate and border gradient sequences to sustain information flow over long step-index ranges for numerically well-behaved objective gradients.The relevant sequences are α_cs[n], α_cu[n], α_cr[n], α_du[n], χ[n], and ψ[n].
  • Scope of analysis: The section treats exploding gradients separately and focuses on assessing the LSTM’s effectiveness in alleviating vanishing gradients during training.The analysis interprets the long-range derivative as the fraction of an impulse-like corrective stimulus that survives.
  • Vanishing-gradient regimes: When the spectral radius of the recurrence term is below unity at every step, error gradients eventually become negligible because the training data contain naturally short-range dependencies.This decay is attributed to data patterns rather than a degenerate system mode.
  • Vanishing-gradient regimes: For long-range dependencies, powers of the recurrence term become negligible as the step gap grows, causing the corresponding gradient contribution to vanish.The analysis evaluates how recurrence factors attenuate error-gradient signals over large l − n.
  • Constant Error Carousel: The Constant Error Carousel avoids attenuation because the state-to-state gradient multiplier is the identity matrix, recirculating error when the state signal saturates at one.If state elements remain fractional, the error gradient naturally decays instead.

VII. EXTENSIONS TO THE VANILLA LSTM NETWORK

The section extends the Vanilla LSTM architecture along three avenues, beginning by combining multiple samples within a small context window instead of using a single sample as input.

  • Extensions to the Vanilla LSTM Network: The Vanilla LSTM architecture is evolved along three avenues, including expansion from a single input sample to multiple samples combined within a small context window.This extension builds on analyses and discussions from Sections II, V, and VI-I.

A. External Input Context Windows · B. Recurrent Projection Layer · C. Controlling External Input with a New Gate

The augmented LSTM introduces non-causal external-input context windows, a recurrent projection layer, and a control input gate. Together, these modifications provide look-ahead context, reduce recurrent value dimensionality, and regulate external-input contribution to data updates.

  • A. External Input Context Windows: External input context windows replace single-sample matrix products with convolutions across the unrolled sequence dimension.The filters use matrix-valued coefficients operating on sequences of input samples.
  • A. External Input Context Windows: Each context-window filter can contain L non-zero matrix-valued terms, so L = 4 uses Wx[0], Wx[−1], Wx[−2], and Wx[−3].The resulting convolution depends on a window of four input samples.
  • A. External Input Context Windows: Non-causal windows expose dependence on future unrolled steps, enabling the system to learn the context surrounding each step.This look-ahead design is motivated by reading forward to interpret an earlier phrase.
  • B. Recurrent Projection Layer: The recurrent projection layer redefines the cell’s value signal as an additional weighted transformation of the Vanilla LSTM cell signal.The intermediate qualifier signal replaces the former value signal, while the new value signal feeds the accumulation signals.
  • B. Recurrent Projection Layer: Allowing dv < ds reduces the dimensionality of the observable value signal and can speed training by shrinking dominant recurrent matrix multiplications.The observable value signal has dimensionality dv, with Wqdr ∈ R^(dv×ds).
  • C. Controlling External Input with a New Gate: The new control input gate addresses the unequal treatment of data-update components, where the readout candidate is throttled but external input receives a full 100% contribution.The modification specifically targets the external-input term in the data update accumulation node.
  • C. Controlling External Input with a New Gate: The control input gate gcx[n] throttles the convolutional external-input signal Wxdu[n] ∗ x[n] before it contributes to the data update accumulation node.The gate is computed by applying the control warping function element by element to its accumulation signal.

D. Augmented LSTM System Equations (“Forward Pass”)

This section assembles the Augmented LSTM forward-pass equations by extending the Vanilla LSTM with a recurrent projection layer, non-causal input context windows, and an input gate. It specifies the adjusted parameter dimensions and combines the cell’s matrix and vector parameters into the complete inference definition.

  • The Augmented LSTM enhances the Vanilla LSTM with a recurrent projection layer, non-causal input context windows, and an input gate.
  • The recurrent projection layer requires adjusted parameter dimensions, including output projection Wqdr ∈ R^{dv×ds}, with dv ≤ ds.The passage also specifies dimensions for the input, recurrent, context, and bias parameters across the cell components.
  • Combining the Augmented LSTM cell’s matrix and vector parameters from Equations 196–214 completes its forward-pass definition.The combined parameter set includes weights and biases for the cell’s augmented components and the recurrent projection.
  • Biases may be implemented by appending a constant 1 to the input vector and increasing the corresponding weight-matrix row dimensions.The schematic omits bias parameters for brevity while retaining this equivalent implementation.

E. Augmented LSTM System Derivatives: Backward Pass · VIII. CONCLUSIONS AND FUTURE WORK

The Augmented LSTM backward pass applies BPTT and the chain rule to derive parameter gradients, which are then aggregated across segments for gradient-descent training. The paper concludes by presenting RNN and LSTM fundamentals through a principled derivation beginning with differential equations and sampled delay differential equations.

  • E. Augmented LSTM System Derivatives: Backward Pass: The Augmented LSTM cell’s K-step BPTT training equations reuse the Vanilla LSTM method and add the objective derivative with respect to the qualifier signal.This added intermediate derivative reflects the projection layer inserted into the cell’s data path.
  • E. Augmented LSTM System Derivatives: Backward Pass: Applying the chain rule to Equations 196–214 and using border and intermediate total partial derivatives yields the Augmented LSTM backward-pass equations.The derivation refers to the Augmented LSTM cell definition and its associated intermediate derivatives.
  • E. Augmented LSTM System Derivatives: Backward Pass: For 0 ≤ l ≤ L − 1 and 0 ≤ n ≤ K − 1, parameter derivatives are arranged to parallel the structure of Θ at step n.The listed derivatives include those with respect to Wxcr[l][n], Wscr[n], Wvcr[n], and b⃗cr[n].
  • E. Augmented LSTM System Derivatives: Backward Pass: The total derivative dE/dΘ for the entire unrolled sequence is computed by Equation 172 and aggregated over a batch of segments.The aggregated derivative is supplied to the Gradient Descent training algorithm to learn Θ.
  • VIII. CONCLUSIONS AND FUTURE WORK: The paper presents RNN and LSTM fundamentals using a principled approach grounded in differential equations from science and engineering.The conclusion states that the canonical RNN formulation is obtained by sampling delay differential equations used to model processes in physics, life sciences, and neural networks.
  • VIII. CONCLUSIONS AND FUTURE WORK: The paper derives the canonical RNN formulation by sampling delay differential equations and proceeds toward the standard RNN formulation by choosing canonical-RNN parameters.The supplied conclusion passage describes these derivational steps but ends before stating the complete standard RNN result.
Loading 1808.03314v10…