Source-linked AI summary
A comparison of LSTM and GRU networks for learning symbolic sequences
Roberto Cahuantzi, Xinye Chen, Stefan Güttel
TL;DR
The paper asks how RNN architecture and hyper-parameters affect memorization of symbolic sequences with different complexity. It generates and trains on such sequences, evaluates forecasts with text-similarity metrics, and compares LSTMs with GRUs. Learning rate and unit count are especially influential, while GRUs perform better on low-complexity sequences and LSTMs on high-complexity sequences.
Problem
The paper investigates how RNN architecture and hyper-parameters relate to the ability to learn symbolic sequences of differing complexity.
Method
The study generates symbolic sequences with quantifiable complexity, trains LSTM and GRU networks, and evaluates forecast accuracy using text-edit metrics including Damerau–Levenshtein and Jaro–Winkler distances.
Results
GRUs outperform LSTMs on low-complexity sequences, whereas LSTMs perform better on high-complexity sequences; learning rate and hidden-unit count are influential hyper-parameters.
Takeaways & Limitations
Single-layer RNNs with moderate unit counts are often sufficient for the considered tasks, while greater depth increases training time without necessarily improving forecast accuracy.
Takeaways & Limitations
When string complexity becomes very high, learning capability may become restricted, with lower mean text similarity and more outlier scattering.
Abstract
from arXiv · showhide
We explore the architecture of recurrent neural networks (RNNs) by studying the complexity of string sequences it is able to memorize. Symbolic sequences of different complexity are generated to simulate RNN training and study parameter configurations with a view to the network's capability of learning and inference. We compare Long Short-Term Memory (LSTM) networks and gated recurrent units (GRUs). We find that an increase in RNN depth does not necessarily result in better memorization capability when the training time is constrained. Our results also indicate that the learning rate and the number of units per layer are among the most important hyper-parameters to be tuned. Generally, GRUs outperform LSTM networks on low-complexity sequences while on high-complexity sequences LSTMs perform better.
1 Introduction
This study examines how RNN hyper-parameters relate to the complexity of symbolic sequences, using empirical comparisons of common architectures. It highlights learning-rate and architecture choices, and reports different GRU-versus-LSTM outcomes across sequence complexities.
- Study focus: The study empirically connects RNN hyper-parameters with the complexity of string sequences being learned and forecasted.It focuses on symbolic sequence learning rather than numerical data.
- Main findings: Learning rate is among the most influential parameters for training RNNs to memorize sequences.Values near 10^-2 were best in the reported setup for training time and forecast accuracy.
- Main findings: For the considered tasks, common single-layer RNNs with moderate capacity, such as around 100 units, are often sufficient.The paper contrasts this simpler setup with more complex global forecasting models whose hyper-parameter relationships are harder to interpret.
- Main findings: GRUs outperform LSTMs on low-complexity sequences, whereas LSTMs perform better on high-complexity sequences.The comparison concerns two widely used recurrent architectures.
- Practical contribution: The study uses symbolic sequences with user-specific complexity to support comparative investigation of RNN models and related parameter choices.Its methods were incorporated into a Python library for generating synthetic symbolic sequences and comparing models.
2 Methodology
The methodology generates symbolic sequences with controlled complexity, trains LSTM and GRU models on repeated seed strings, and evaluates forecasts using text-similarity measures.
- Sequence generation: Symbolic seed strings are generated with selectable target complexity and numbers of distinct symbols.Complexity is estimated using LZW compression, whose complexity is defined as the length of the associated LZW array.
- Training, test, and validation data: Each seed string is repeated to a minimum length, with trailing characters reserved for validation and sliding windows producing training input-output pairs.The input array X contains sequences of length n, while y contains the next symbol after each sequence; the data is split 95% for training and 5% for testing.
- Sequence generation: LZW traverses strings while building a substring dictionary and replacing recognized substrings with dictionary indices.For "ABABCBABAB", the resulting array is [1, 2, 4, 3, 5, 8], giving an LZW complexity of 6.
- Recurrent neural networks: The study compares Long Short-Term Memory cells and Gated Recurrent Units as the two recurrent architectures.LSTMs use forget, input, and output gates; GRUs use update and reset gates and fewer parameters.
- Text similarity metrics: Forecast accuracy is assessed with Damerau–Levenshtein and Jaro–Winkler text distances because symbolic sequences are non-Euclidean.Damerau–Levenshtein counts edit operations, while Jaro–Winkler is based on similarity and accounts for matching characters, transpositions, and common prefixes.
3 Results
The experiments examine how learning rate, network depth, and architecture affect RNN training time and forecast accuracy across string complexities. Learning rate and hidden-unit count are influential, while added depth offers limited accuracy gains relative to its training-time cost; GRUs are faster on low-complexity strings, whereas LSTMs are better suited to high-complexity strings.
- Initial parameter test: A learning rate of 0.01 led to the smallest training times across all considered string complexities and both stopping criteria.This result was observed in the initial parameter study and summarized in Figure 5.
- Network depth: Single-layer networks offered the most successful balance of forecast accuracy and training time when the total number of units was approximately held constant.Adding hidden layers slightly improved forecast accuracy but increased training time enough that the depth increase was not justified.
- Hyper-parameters: Learning rate and the number of hidden units were among the most influential hyper-parameters for RNN effectiveness in the initial parameter test.The test trained 3,239 RNNs over five runs using 12 seed strings.
- Low-complexity strings: GRUs trained faster than LSTMs on low-complexity seed strings while achieving comparable forecast accuracy.Median training time was 19.72 seconds for GRUs versus 37.19 seconds for LSTMs; both models had median JW and DL distances of 1.0.
- High-complexity strings: LSTMs were better suited than GRUs for high-complexity strings, with median training times of 12.53 seconds and 22.84 seconds, respectively.For high-complexity strings, LSTMs were faster to train than GRUs for similar forecast performance.
4 Discussion
The discussion finds that LSTMs generally perform better on high-complexity strings, although one high-complexity figure reports similar forecast accuracy and faster LSTM training. Increasing string complexity can eventually restrict learning quality for both architectures.
- LSTMs generally performed better than GRUs on high-complexity strings, reversing the low-complexity pattern.
- High-complexity seed strings produced similar forecast accuracy for LSTM and GRU, while LSTMs trained faster.
- Very high string complexity may restrict learning, manifested by lower mean text similarity and greater outlier scattering.
- As string complexity increases, both networks show declining mean text similarity, indicating degraded forecasting quality.The trend was observed with optimized hyper-parameters and training capped at 999 epochs.