Source-linked AI summary
On Extended Long Short-term Memory and Dependent Bidirectional Recurrent Neural Network
Yuanhang Su, C. -C. Jay Kuo
TL;DR
The paper investigates memory decay in SRN, LSTM, and GRU cells, then proposes ELSTM and DBRNN to improve sequence modeling. It evaluates the resulting system on language modeling, POS tagging, and dependency parsing.
Problem
The paper addresses limited understanding of memory decay in recurrent cells and the need for effective sequence learning systems.
Method
The authors define and analyze RNN memory, propose ELSTM to extend cell memory, and introduce DBRNN as a multi-task bidirectional recurrent architecture.
Results
The proposed ELSTM-DBRNN system is evaluated on language modeling, POS tagging, and dependency parsing.
Takeaways & Limitations
The paper presents ELSTM and DBRNN as proposed solutions for improving recurrent sequence learning across several language tasks.
Takeaways & Limitations
The SRN analysis typically assumes an initial internal state c0 = 0 and focuses on the mathematically tractable memory-decay case.
Abstract
from arXiv · showhide
In this work, we first analyze the memory behavior in three recurrent neural networks (RNN) cells; namely, the simple RNN (SRN), the long short-term memory (LSTM) and the gated recurrent unit (GRU), where the memory is defined as a function that maps previous elements in a sequence to the current output. Our study shows that all three of them suffer rapid memory decay. Then, to alleviate this effect, we introduce trainable scaling factors that act like an attention mechanism to adjust memory decay adaptively. The new design is called the extended LSTM (ELSTM). Finally, to design a system that is robust to previous erroneous predictions, we propose a dependent bidirectional recurrent neural network (DBRNN). Extensive experiments are conducted on different language tasks to demonstrate the superiority of the proposed ELSTM and DBRNN solutions. The ELTSM has achieved up to 30% increase in the labeled attachment score (LAS) as compared to LSTM and GRU in the dependency parsing (DP) task. Our models also outperform other state-of-the-art models such as bi-attention and convolutional sequence to sequence (convseq2seq) by close to 10% in the LAS. The code is released as an open source (https://github.com/yuanhangsu/ELSTM-DBRNN)
1. Introduction
The paper studies memory limitations in recurrent cells and proposes ELSTM to extend memory, then introduces DBRNN to combine encoder-decoder and bidirectional recurrence for language tasks.
- 1. Introduction: LSTM and GRU were designed to improve RNN memory length and address vanishing or exploding gradients, but their memory decay had not been thoroughly analyzed.
- 1. Introduction: The work analyzes how SRN, LSTM, and GRU retain sequence information and distinguishes memory analysis from training-time gradient analysis.Memory is defined through dependencies between prior sequence elements and the current output.
- 1. Introduction: The proposed extended long short-term memory (ELSTM) is designed to extend a recurrent cell’s memory length based on the preceding memory analysis.
- 1. Introduction: The dependent bidirectional recurrent neural network (DBRNN) jointly exploits encoder-decoder and bidirectional RNN models in a new multi-task architecture.
- 1. Introduction: Experiments evaluate the ELSTM-DBRNN system on language modeling, part-of-speech tagging, and dependency parsing.
2. Memory Analysis of SRN, LSTM and GRU
This section defines RNN memory as the mapping from prior inputs to current outputs and analyzes memory behavior in SRN, LSTM, and GRU cells.
- 2. Memory Analysis of SRN, LSTM and GRU: RNN memory is defined as the function mapping elements of an input sequence to the current output, including how that mapping occurs.
- 2.1. Memory of SRN: An SRN retains semantic sequential patterns through dependencies on preceding inputs, but its memory length is limited.
- 2.1. Memory of SRN: Under the analyzed decay condition, an earlier SRN input’s contribution to the current output decays exponentially with the sequence distance t − k.
- 2.2. Memory of LSTM: The LSTM cell uses input, output, and forget gates together with a constant error carousel module.
- 2.2. Memory of LSTM: Special cases exist in which LSTM memory lasts longer than SRN memory regardless of the SRN parameter choice.
a LSTM whose memory length is longer than SRN for all possible
The analysis finds that LSTM and GRU memory responses inevitably diminish with sequence distance because their gate multiplications decay. It also shows that GRU and LSTM have no fundamental difference in this memory behavior.
- SRN comparison: A comparable SRN design would require conditions that can easily cause memory explosion.
- GRU–LSTM comparison: GRU's update gate and LSTM's forget gate play the same role, revealing no fundamental difference between the cells.
- Memory decay: LSTM and GRU memory responses to earlier inputs inevitably diminish as their temporal distance increases.The decay arises from multiplication terms introduced by the forget and update gates.
- Motivation: Memory decay can significantly affect complex language tasks that require long memory responses, such as sentence parsing.
3. Extended Long Short-Term Memory (ELSTM)
ELSTM addresses fast memory decay by introducing trainable scaling factors that adaptively adjust the impact of earlier inputs. These factors are designed to provide longer memory than standard LSTM.
- ELSTM design: ELSTM introduces trainable scaling factors to compensate for fast decay in input responses.The scaling factors can increase or decrease the impact of each input in the sequence.
- ELSTM design: Each ELSTM scaling vector adaptively adjusts the impact of an input at its sequence position.
- Memory analysis: The proposed analysis aims to show that ELSTM has longer memory than LSTM.
exists an ELSTM whose memory is longer than LSTM for all choices
ELSTM extends LSTM memory through trainable, periodic scaling factors that compensate for memory decay, while experiments show stronger retention on longer sequences.
- ELSTM design: ELSTM’s parameter count depends on maximum input length, making model size uncontrollable without a fixed periodic bound.The paper addresses this by choosing a fixed Ts smaller than T as the upper bound on scaling factors.
- ELSTM design: Trainable scaling factors compensate for recurrent memory decay and give ELSTM a design advantage over LSTM.The factors are periodic, with the number of factors tied to the memory length required by the task.
- Memory experiment: For T = 10, ELSTM responds more selectively to a marked element than LSTM, with a smaller response at the following position.The learned scaling factor rises at the marked position and compensates near sequence boundaries.
- Memory experiment: For T = 60, LSTM fails to strongly remember the marked element, whereas ELSTM retains a strong response, especially near the sequence middle.The scaling factor compensates in the first half and learns the training-set average importance of positions, but cannot change adaptively after training.
4. Dependent BRNN (DBRNN) Model
DBRNN combines bidirectional processing with an encoder-decoder structure to use full input context while reducing vulnerability to previous erroneous predictions.
- Motivation: DBRNN is proposed as a macro RNN that combines bidirectional RNN and encoder-decoder designs.The design is motivated by BRNN’s full-sequence context and encoder-decoder alignment, while addressing forward-path prediction errors.
- BRNN and encoder-decoder: BRNN uses forward and backward information so each output can access the entire input sequence, including future inputs.This can help when the current output depends on following input elements.
- BRNN and encoder-decoder: The encoder-decoder encodes an input sequence into its final hidden state and then generates an output sequence using a decoder.Its decoder begins from a start symbol, feeds previous outputs forward, and stops upon generating a stopping symbol.
- DBRNN architecture: DBRNN trains three objectives for forward, backward, and bidirectional predictions, with errors backpropagated through the corresponding branches and entire model.This multi-task design is intended to make the model robust to previous erroneous predictions.
- DBRNN architecture: DBRNN stacks lower and upper BRNN branches, feeding concatenated forward and backward lower-branch outputs into the upper branch.The upper forward and backward paths are trained with original and reversed target sequences, whose predictions are then combined.
- DBRNN properties: DBRNN is presented as better aligned than encoder-decoder models and more robust to previous erroneous predictions through bidirectional decoding.The paper also describes it as a generic SISO solution not restricted to dependency parsing.
5. Experiments
Experiments compare RNN macro-models and cells across POS tagging, language modeling, and dependency parsing, with ELSTM and DBRNN generally strongest on complex tasks.
- Experimental setup: The evaluation covers POS tagging, language modeling, and dependency parsing using accuracy, perplexity, UAS, and LAS.The tasks differ in memory demands, with language modeling and dependency parsing requiring longer memory than POS tagging.
- Cell comparison: ELSTM outperforms LSTM and GRU in most macro-models, with especially large gains on language modeling and dependency parsing.ELSTM also outperforms the bi-attention model designed specifically for dependency parsing and remains equal or better on POS tagging.
- Dependency parsing: DBRNN reaches 51.38/39.71, 52.23/37.25, and 61.35/43.32 on the reported test-result rows.These values are presented as UAS/LAS pairs in the DBRNN row.
- Scaling-factor study: For scaling-factor counts, one performs best for POS tagging, three for language modeling, and larger values are preferred for seq2seq models.The authors relate these settings to task-specific memory requirements and recommend estimating memory length before hyperparameter selection.
- Macro-model comparison: DBRNN achieves the best language-modeling performance across macro-models and outperforms BRNN and seq2seq on POS tagging and dependency parsing across cell types.The authors interpret these results as evidence of DBRNN’s robustness, while noting possible overfitting in other cases.
- Non-RNN comparison: ELSTM-based models learn much faster than the CNN-based ConvSeq2seq model while using fewer parameters.The comparison is reported in the discussion of Table 8.
6. Conclusion and Future Work
The paper concludes that ELSTM improves recurrent-cell memory for complex language tasks, while DBRNN outperforms BRNN and seq2seq models. It also identifies deeper architectures and broader ELSTM applications as future work.
- ELSTM enhances the memory capability of an RNN cell to address insufficient memory for complicated language tasks such as dependency parsing.
- The proposed DBRNN combines merits of BRNN and encoder-decoder models.
- ELSTM outperforms other RNN cell designs by a significant margin on complex language tasks.
- DBRNN is superior to BRNN and seq2seq models for both simple and complex language tasks.
- The conclusion leaves applying ELSTM to deeper RNNs and making DBRNN deeper and better for future study.
Appendix A: More Experimental Results
The appendix reports additional training-perplexity comparisons for LSTM and ELSTM models on the POS-tagging task.
- Training perplexity is examined as a function of training steps under alternative input settings.
- The appendix compares training perplexity across different models using LSTM and ELSTM cells for the POS-tagging task.