Source-linked AI summary
Recurrent Neural Networks (RNNs): A gentle Introduction and Overview
Robin M. Schmidt
TL;DR
RNNs underpin important sequence-processing applications, but understanding their training challenges and architectural advances requires a concise grounding in core concepts. This paper surveys RNN fundamentals, BPTT, LSTMs, encoder-decoder models, attention, Transformers, and Pointer Networks, while directing readers to original papers for deeper treatment.
Problem
RNNs support language modeling, speech recognition, image description, and video tagging, creating a need to understand their underlying concepts and training techniques.
Method
The paper provides a conceptual overview of RNN frameworks, BPTT, gradient problems, LSTMs, bidirectional and deep RNNs, encoder-decoder models, attention, Transformers, and Pointer Networks.
Results
The overview covers foundational RNN concepts alongside more recent advances and recommends cited original papers for broader understanding.
Takeaways & Limitations
Readers can use the overview to understand concepts appearing in recent publications and identify further reading for topics requiring greater depth.
Takeaways & Limitations
Multi-headed attention and its mathematical formulations are not explained in detail, and most topics are treated conceptually rather than at implementation depth.
Abstract
from arXiv · showhide
State-of-the-art solutions in the areas of "Language Modelling & Generating Text", "Speech Recognition", "Generating Image Descriptions" or "Video Tagging" have been using Recurrent Neural Networks as the foundation for their approaches. Understanding the underlying concepts is therefore of tremendous importance if we want to keep up with recent or upcoming publications in those areas. In this work we give a short overview over some of the most important concepts in the realm of Recurrent Neural Networks which enables readers to easily understand the fundamentals such as but not limited to "Backpropagation through Time" or "Long Short-Term Memory Units" as well as some of the more recent advances like the "Attention Mechanism" or "Pointer Networks". We also give recommendations for further reading regarding more complex topics where it is necessary.
1 Introduction & Notation
RNNs process sequential data while retaining information from previous inputs, distinguishing them from feedforward networks. The section introduces their hidden-state notation and explains how BPTT unfolds recurrent computation for training.
- RNN fundamentals: RNNs detect patterns in sequences such as text, handwriting, genomes, time series, image patches, and video-related tasks.Their recurrent structure also supports language modelling, speech recognition, image-description generation, and video tagging.
- RNN fundamentals: Unlike feedforward networks, RNNs transmit information through cycles and incorporate previous inputs X0:t−1 alongside the current input Xt.This difference is visualized in Figure 1.
- Notation: The hidden state Ht and input Xt are represented with sample, input, and hidden-unit dimensions, alongside input-to-hidden, hidden-to-hidden, and bias parameters.An activation function, usually sigmoid or tanh, prepares the computation for backpropagation.
- Training with BPTT: BPTT adapts backpropagation to RNNs by unfolding the recurrent network into a traditional feedforward network.The forward pass computes hidden and output states step by step, while the loss sums per-time-step losses.
- Training with BPTT: Gradients are computed for the input-to-hidden, hidden-to-hidden, and hidden-to-output weight matrices using the chain rule across time steps.Because each hidden state depends on the preceding state, the resulting derivatives include powers of the hidden-state transition matrix.
- Training with BPTT: Truncated BPTT limits gradient flow to a computationally convenient moving window of past time steps.This bounds the effective number of unfolded hidden layers and avoids storing arbitrarily large powers of Whh.
3 Problems of RNNs: Vanishing & Exploding Gradients
RNN training can suffer from vanishing or exploding gradients because repeated matrix multiplications across long sequences attenuate or amplify gradients. LSTMs were introduced to address the vanishing-gradient problem and outperformed traditional RNNs on a variety of tasks.
- Gradient instability: Vanishing and exploding gradients are key problems in traditional RNNs.They arise during repeated matrix multiplication over potentially very long sequences.
- Vanishing gradients: Values below 1 in the repeated matrix multiplication can make gradients decrease until they vanish, removing contributions from states far earlier in the sequence.This limits the influence of distant history on the current time step.
- Exploding gradients: Values above 1 can cause exploding gradients, making weights change heavily during training.The same recurrent multiplication process can therefore produce either vanishing or exploding behavior.
- Motivation for LSTMs: LSTMs were introduced to handle the vanishing-gradient problem and outperformed traditional RNNs on a variety of tasks.Their design is discussed in the following section.
4 Long Short-Term Memory Units (LSTMs)
LSTMs address vanishing gradients by using gated cells and a more constant error pathway, allowing RNNs to learn across much longer sequences. Their gates regulate memory updates, and the resulting cell state produces the hidden state.
- LSTM motivation: LSTMs use gated cells and a more constant error to let RNNs learn over more than 1000 time steps.They store information outside the traditional neural-network flow.
- Gated cells: The output gate Ot reads entries from the cell, the input gate It reads data into it, and the forget gate Ft resets cell content.These gates use sigmoid activation to produce values between 0 and 1.
- Memory candidate: The candidate memory cell ˜Ct uses tanh activation and its own weights and biases to propose new memory content.Its output lies between −1 and 1.
- Memory update: The new memory cell Ct combines retained old content Ct−1 with candidate content through the forget and input gates.The operation uses element-wise multiplication, denoted by ⊙.
- Hidden state: The LSTM then computes hidden states Ht from the updated framework, with tanh constraining each hidden-state element between −1 and 1.The full framework is illustrated in Figure 11 in the appendix.
5 Deep Recurrent Neural Networks (DRNNs)
Deep Recurrent Neural Networks stack recurrent layers so each layer passes hidden states across time while higher layers receive representations from lower layers.
- A deep RNN is constructed by stacking L recurrent hidden layers of any RNN type.
- Each hidden state is passed both to the next time step in its current layer and to the subsequent layer.
- The first layer uses the standard recurrent hidden-state computation, whereas later layers treat the preceding layer’s hidden state as input.
- The output uses only the hidden state from the final layer and has shape R^n×o, where o is the number of outputs.
6 Bidirectional Recurrent Neural Networks (BRNNs)
Bidirectional RNNs address tasks requiring future context by processing sequences in both directions and combining the resulting hidden states.
- This look-ahead property is useful when information after a gap is significant, such as in sentence completion or related sequence tasks.
- Bidirectional RNNs add a backward hidden layer so predictions can incorporate information from both preceding and following sequence elements.The forward direction starts at the first element, while the backward direction starts at the last.
- The forward and backward hidden states are computed separately using two sets of weight matrices.
- The output concatenates the forward and backward hidden-state matrices before applying the output transformation.The output has shape R^n×o, where o is the number of outputs.
7 Encoder-Decoder Architecture & Sequence to Sequence (seq2seq)
Encoder-decoder networks encode an input sequence into a state and decode that state into an output sequence; seq2seq uses RNNs for both stages. Fixed-length context vectors can bottleneck long-sequence modeling, motivating attention-based approaches.
- Encoder-Decoder Architecture: An encoder network converts the input into a state, and a decoder network converts that state into an output.The state is usually represented as a vector or tensor.
- Sequence to Sequence (seq2seq): Seq2seq models generate sequence outputs from sequence inputs by passing the encoder’s hidden state to the decoder.They can map an input sequence of length n to an output sequence of length m, where n ≠ m is allowed.
- Sequence to Sequence (seq2seq): Seq2seq models are applied to tasks including translation, voice-enabled devices, and video-data labeling.
- Sequence to Sequence (seq2seq): The encoder processes sequence elements with an RNN, whose final hidden state forms a context vector used to initialize the decoder.The encoder and decoder can use LSTMs or GRUs, and the decoder predicts one output sequence element at each time step.
- Limitations and Attention: The fixed-length context vector is a bottleneck because it must contain all necessary information from a source sentence, especially for long sequences.Attention-based approaches were introduced to address this bottleneck.
8 Attention Mechanism & Transformer
Attention mechanisms connect outputs to relevant source positions, helping sequence models handle long inputs. Transformers extend this idea with self-attention, positional encoding, and parallel processing without recurrent units.
- Attention Mechanism: Attention represents relationships between sequence elements in a matrix, with lighter entries indicating higher correlation.The same mechanism can operate across two sentences or within one sentence as self-attention.
- Attention Mechanism: Attention replaces the fixed final encoder state with access to all source hidden states, addressing the fixed-length bottleneck for long sequences.The encoder passes its entire sequence of hidden states to the decoder, which computes context vectors for successive outputs.
- Attention Mechanism: In attention-based seq2seq models, each context vector sums encoder hidden states weighted by alignment scores that normalize to 1.Each alignment score links an input position to an output position and indicates how strongly that source state should be considered.
- Transformer: The Transformer uses self-attention and positional encoding to parallelize sequence processing without recurrent network units.Its encoder and decoder stacks include feed-forward layers, encoder-decoder attention, skip connections, and layer normalization.
- Transformer: Multi-headed attention jointly attends to different representation subspaces and positions by processing chunks in parallel before concatenating results.The paper notes that the detailed design and mathematical formulation are deferred to the original Transformer paper.
- Transformer: The Transformer’s final linear and softmax layers convert decoder vectors into vocabulary-sized logits and word probabilities.The probabilities sum to 1, allowing selection of a word from the learned training vocabulary.
9 Pointer Networks (Ptr-Nets)
Pointer Networks adapt attention-based seq2seq models so outputs point to elements of the input rather than selecting from a fixed output dictionary.
- Pointer Networks: Pointer Networks do not fix the discrete output categories in advance; they generate pointers to elements of the input sequence.This changes the output from a generated sequence over a predefined dictionary to a succession of input positions.
- Applications: Pointer Networks were used to solve combinatorial optimization problems including planar convex hulls and Delaunay triangulation.These examples are reported from the cited Pointer Networks work.
- Pointer Networks: Pointer Networks use additive attention between states and normalize the resulting scores with softmax to model output probabilities.The paper describes this formulation in Equation 29.
- Pointer Networks: In Ptr-Nets, attention responds to input positions rather than blending encoder states into outputs with attention weights.The paper states that the output therefore depends on positions, not input content.
10 Conlusion & Outlook
The paper provides a conceptual introduction to RNN fundamentals and several extensions, while directing readers to original publications for deeper implementation details and recent applications.
- Conclusion: The overview covers RNN frameworks, BPTT, traditional RNN problems, LSTMs, deep and bidirectional RNNs, encoder-decoder models, seq2seq, attention, Transformers, and Pointer Networks.The authors present these topics as fundamentals and recent advances in recurrent-network research.
- Outlook: Most topics are covered conceptually rather than through detailed implementation specifications, so the authors recommend consulting cited original papers.They also recommend recent publications that use the presented concepts.
- Outlook: StarCraft II multi-agent reinforcement learning is offered as a practical example using several presented concepts, including LSTMs, Transformers, and Pointer Networks.The paper recommends this work for readers seeking an applied setting beyond the overview’s theory.
A Visual Representation of LSTMs
The visual walkthrough constructs LSTM components and then traces attention-based seq2seq decoding step by step, from encoder states to successive outputs.
- LSTM Construction: The LSTM walkthrough consecutively constructs the full Long Short-Term Memory architecture from its component computations.The referenced figures cover input, forget, and output gates, candidate memory cells, memory cells, and hidden states.
- Attention Decoder: Attention-based seq2seq passes all encoder hidden states to the decoder instead of only the encoder’s final hidden state.The walkthrough illustrates translation of “I am a student” into French.
- Attention Decoder: At the first decoder step, the model produces a new hidden state and discards the corresponding output before computing attention.The new hidden state is used with encoder states in the next attention step.
- Attention Decoder: The decoder uses the current hidden state and encoder states to score positions, then combines them into a context vector through softmax-weighted states.Higher scores preserve more contribution from the associated encoder state.
- Attention Decoder: The context vector is concatenated with the decoder hidden state and passed through a jointly trained feed-forward network to produce the current output.The illustrated first output represents the word “I”.
- Attention Decoder: On later iterations, the previous hidden-state output replaces the end token, while the same attention and decoding procedure is repeated.The walkthrough shows a different encoder state receiving the highest attention score at the next step; additional steps are omitted.
C Visual Representation of Positional Encodings used in the Transformer
The Transformer uses trigonometric positional encodings to represent sequence positions, with similar encodings for nearby words and more distinct encodings for distant words. This encoding supports the model’s positional representation but is not presented as the main contribution.
- Trigonometric functions with different frequencies encode the positions of words such as X1, X2, and X3.Each word receives a pattern across multiple frequency curves.
- Nearby words receive more similar encodings, whereas distant words receive more different encodings.The encoding represents proximity between sequence elements.
- The positional encoding choice is relevant for understanding the Transformer theoretically but is not its main contribution.The passage states that positional encoding boosts performance.