Source-linked AI summary
Working Memory Connections for LSTM
Federico Landi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara
TL;DR
LSTMs mitigate exploding and vanishing gradients with gates, but their memory cells cannot directly influence gate dynamics. The paper introduces Working Memory Connections, which provide protected, learnable access from the cell state to the gates. Across multiple tasks, WMCs improve LSTM performance and training stability, motivating broader evaluation of cell-to-gate connections.
Problem
LSTM cell states contain useful long-term information but traditionally cannot directly influence gate dynamics, while earlier cell-to-gate connections showed inconsistent benefits.
Method
Working Memory Connections add a protected learnable nonlinear projection of the memory cell into the classical LSTM gates.
Results
Working Memory Connections improve LSTM performance across toy long-dependency problems, language modeling, and image captioning, while providing more stable training than peephole connections.
Takeaways & Limitations
The results indicate that information stored in the LSTM cell is useful for regulating the gate structure.
Takeaways & Limitations
The paper identifies direct unprotected cell-state exposure as a source of unstable training dynamics and notes that wider task coverage remains future work.
Abstract
from arXiv · showhide
Recurrent Neural Networks with Long Short-Term Memory (LSTM) make use of gating mechanisms to mitigate exploding and vanishing gradients when learning long-term dependencies. For this reason, LSTMs and other gated RNNs are widely adopted, being the standard de facto for many sequence modeling tasks. Although the memory cell inside the LSTM contains essential information, it is not allowed to influence the gating mechanism directly. In this work, we improve the gate potential by including information coming from the internal cell state. The proposed modification, named Working Memory Connection, consists in adding a learnable nonlinear projection of the cell content into the network gates. This modification can fit into the classical LSTM gates without any assumption on the underlying task, being particularly effective when dealing with longer sequences. Previous research effort in this direction, which goes back to the early 2000s, could not bring a consistent improvement over vanilla LSTM. As part of this paper, we identify a key issue tied to previous connections that heavily limits their effectiveness, hence preventing a successful integration of the knowledge coming from the internal cell state. We show through extensive experimental evaluation that Working Memory Connections constantly improve the performance of LSTMs on a variety of tasks. Numerical results suggest that the cell state contains useful information that is worth including in the gate structure.
1. Introduction
LSTMs use gates to regulate information flow and address exploding and vanishing gradients, but their memory cells traditionally cannot directly influence those gates. The paper introduces Working Memory Connections, which add protected, learnable access from the cell state to the gates and improve stability and performance across tasks.
- LSTMs use gates to control information flow into and from the memory cell and regulate forgetting.
- Working Memory Connections add a protected connection from the memory cell to the gates through a learnable projection.The protection mechanism prevents the cell state from being exposed directly.
- The proposed modification lets long-term cell memory help control the network’s short-term evolution without assumptions about the underlying task.
- Earlier peephole connections directly linked the cell to gates, but reported mixed results and were not widely adopted.
- Directly exposing the internal state without protection produces unstable training dynamics that compromise final performance.
- Working Memory Connections are evaluated on long-dependency toy problems, language modeling, and image captioning.
2. Related Work
Related work positions LSTM as a gated RNN for mitigating exploding and vanishing gradients, while contrasting it with simplified recurrent models, optimization-specific methods, GRUs, and self-attention architectures.
- LSTM uses gating mechanisms to mitigate exploding and vanishing gradients in recurrent sequence modeling.
- Liu et al. simplify recurrent blocks to two gates using long-term cell content, but neglect the LSTM output.The passage states that this may suit simple tasks but is unlikely to generalize to more complex settings.
- Arpit et al. stabilize LSTM training by modifying gradient paths with a stochastic algorithm specific to LSTM optimization.
- GRU is described as a popular LSTM variant featuring a coupling mechanism between input and forget gates.
- Transformer-based self-attention models achieve state-of-the-art performance across many tasks, while RNNs require fewer parameters and operations to run.
3. Proposed Method
Working Memory Connections let the LSTM memory cell influence its gates through protected nonlinear projections. The design addresses instability in direct cell-to-gate connections while preserving more responsive control of output dynamics.
- Working Memory Connections: The cell state provides information about previous inputs that may be absent from the hidden output, especially when the output gate saturates toward zero.This connection can let the block learn a mapping between its internal state and gate values.
- Working Memory Connections: Working Memory Connections expose the memory cell to LSTM gates through recurrent weights and a nonlinear activation function.The cell-state contribution is applied through a protection mechanism in the input, forget, and output gates.
- Working Memory Connections: The output-gate connection uses c_t rather than c_t−1, enabling more responsive control of the LSTM block’s output dynamics.Separate nonlinear projections are used for the three gate types.
- Advantages of Working Memory Connections: Unlike direct peepholes, the proposed design bounds the cell-state contribution with a hyperbolic tangent, reducing unwanted gate saturation and unstable weight updates.The transformed cell-state term lies in [−1, 1], while the cell itself can grow with the number of recurrent steps.
- Advantages of Working Memory Connections: Direct peephole connections can push sigmoid gates into saturation and produce recurrent-weight gradients that grow linearly with the cell state.The resulting updates may become unstable, and the gates can be flooded after only a small number of time steps.
- Advantages of Working Memory Connections: Working Memory Connections show more regular cell-state behavior than peepholes and can prevent uncontrolled growth of the memory cell.The paper’s formal analysis is presented for the input gate and generalized to the forget and output gates.
4. Experiments and Results
Experiments compare LSTM-WC with vanilla and peephole LSTM across toy problems, digit recognition, language modeling, and image captioning. Working Memory Connections generally improve performance, especially on longer sequences, while also supporting faster convergence in several tasks.
- Experimental scope: The experiments cover adding, copying, sequential and permuted MNIST, character-level language modeling, and image captioning against vanilla and peephole LSTM baselines.The evaluation spans toy problems with long-term dependencies and real-world sequence applications.
- Adding and copying tasks: For T = 200 in the copying task, WMCs outperform peephole LSTM and vanilla LSTM by around +25% and +40%, respectively.The reported gains are more evident for longer sequences and WMCs also improve convergence time.
- Sequential and permuted MNIST: LSTM-WC exceeds competing architectures on sequential MNIST, reaching above 92% accuracy by epoch 50 while vanilla and peephole LSTM remain around 65% and 82%.On permuted MNIST, all models achieve good final results, with LSTM-WC still reported as the best option.
- Sequential and permuted MNIST: +0.47% and +1.03% are the reported LSTM-WC improvements over classic LSTM on sequential and permuted MNIST, respectively.LSTM-WC achieves the best results even when vanilla and peephole LSTM use increased hidden size and more than twice as many parameters.
- Language modeling: On PTB character-level language modeling, LSTM-WC outperforms vanilla and peephole LSTM by 0.035 and 0.041 BPC for TPTB = 150 and TPTB = 300.Lower BPC denotes better performance, and the advantage is reported when comparing models with fixed parameter counts.
- Image captioning: On image captioning, LSTM-WM improves vanilla LSTM by 2.0 CIDEr points without attention and 0.8 CIDEr points with attention over image regions.The comparison uses both Show and Tell and Up-Down captioning models and reports gains across standard captioning metrics.
5. Discussion
Working Memory Connections let LSTM gates access cell-state information, improving gate control and training stability relative to vanilla LSTM and peephole connections. Benefits may be smaller when cell access is unnecessary or in more complex stacked and compound networks.
- 5. Discussion: Working Memory Connections give LSTM gates access to the cell state, enabling more precise control of information flow.Traditional multiplicative gates cannot access the state of the same cell.
- 5. Discussion: WMCs provide increased training stability compared to peephole connections, whose unsquashed cell-state projections can produce unstable updates.This protection mechanism helps WMCs improve performance even when peepholes fail.
- 5. Discussion: WMCs combine the theoretical benefits of peephole connections with the training stability and versatility of vanilla LSTM.
- 5. Discussion: WMCs may provide little benefit when tasks do not require access to memory-cell content, whereas peepholes may still hinder learning through unstable updates.
- 5. Discussion: WMC benefits may become less significant in stacked LSTMs and compound deep networks, although image-captioning experiments show a small existing improvement.The influence of WMCs in other complex multimodal tasks remains worth investigating.
6. Conclusion
The conclusion presents WMCs as a way to let LSTM cell states influence gate dynamics, addressing a current limitation of LSTMs. The proposed design performs noticeably better than vanilla LSTM, with the improvement attributed to more stable training dynamics.
- 6. Conclusion: Working Memory Connections address the limitation that LSTM cell states cannot directly influence gate dynamics.
- 6. Conclusion: WMCs provide an efficient way to use intra-cell knowledge inside the LSTM network.
- 6. Conclusion: WMCs perform noticeably better than vanilla LSTM, with the improvement formally motivated by more stable training dynamics.
- 6. Conclusion: The authors motivate further study of WMC efficacy across an even wider set of tasks.