Time Series Forecasting Using LSTM Networks: A Symbolic Approach

1 INTRODUCTION

• The introduction notes that no reliable black-box forecasting method is available without manual preprocessing and intensive tuning, while reliability is complicated by many forecast-accuracy measures.Source: §1 · p. 1
• Raw-value machine-learning forecasting can require computationally demanding training and many hyperparameters, and can be sensitive to random-weight initialization.Source: §1 · p. 1
• On a collection of time series, ABBA-LSTM models are more easily trained while achieving similar forecast performance to raw LSTM models.Source: §1 · p. 1
• The paper reports that ABBA-LSTM reduces hyperparameter sensitivity and the need for linear trend removal, and can yield forecasts more faithful to historical behavior.Source: §1 · p. 1

2 SYMBOLIC REPRESENTATION

• ABBA adaptively chooses the symbolic sequence length and alphabet size, then converts the time series through compression and digitization.Source: §2 · p. 2
• Compression partitions the series into pieces and approximates each piece by a straight line encoded as its length and increment.Source: §2 · p. 2
• Digitization clusters the length-increment tuples into symbols; inverse digitization, quantization, and inverse compression convert the symbols back to numeric values.Source: §2 · p. 2
• The proposed patch represents each cluster by its mean pieces: pieces are interpolated or extrapolated to a common average length, averaged into a smooth shape, and stitched in symbol order.Source: §2 · p. 2
• For the illustrated sine wave, ABBA reduces 2,000 values to 27 symbols using four symbols, with a unique symbol for the first segment.Source: Figure 1 · p. 2
• The patched reconstruction uses average shapes from the raw series and is presented as visually more appealing than the standard reconstruction.Source: §2 · p. 2

3 RECURRENT NEURAL NETWORKS

• Unlike a traditional network that ignores temporal dependencies and scales weights with sequence length, an RNN processes elements sequentially with a fixed number of weights.The text motivates LSTMs by the vanishing-gradient problem in deep recurrent networks.Source: §3 · pp. 2–3

3.1 The structure of a single LSTM cell

• An LSTM cell carries hidden and cell states forward, while forget, input, and output gates regulate cell-state retention, updates, and hidden-state exposure.Both states leave the cell at time t and feed back into it at time t + 1.Source: §3.1 · pp. 3–4

3.2 A layer of LSTM cells

• A layer concatenates LSTM cells with separate parameters, whose gate computations can be evaluated together using stacked matrices and element-wise operations.The layer's input and recurrent weights and biases are stacked into compatible matrices and vectors.Source: §3.2 · p. 4

3.3 A multi-layer LSTM network

• In a multi-layer network, hidden states pass sequentially to the next layer, layer widths may differ, and a final layer can map top-layer outputs to time-series values.The final mapping is described as an example for time-series forecasting.Source: §3.3 · p. 5

DATA

• For time-series forecasting, training examples come from a single series without canonical input or output lengths, so sequential-data handling and forecast production require specific choices.The section introduces sequential input, stateful versus stateless training, and forecast generation as those choices.Source: DATA · p. 5

4.1 Feeding in sequential data

• LSTM training feeds an input sequence through repeated applications of the same function S; backpropagation gives each weight in S one additive update per time step.For sequence length ℓ, each weight receives ℓ updates; earlier memory comes from the initial cell and hidden states.Source: §4.1 · p. 5

4.2 ‘stateful’ vs ‘stateless’ training

• Stateless training resets initial states for each training example, allowing examples to be shuffled and mini-batches to be used; stateful training carries states forward between examples.Stateful training preserves chronological order by partitioning examples into ℓ groups whose members stay ordered.Source: §4.2 · p. 6

4.3 Producing forecasts

• Iterated forecasting reuses one one-step function, feeding earlier predictions back as inputs; it avoids specifying k in advance but can accumulate forecast errors.The method makes successive predictions using the previously forecast values.Source: §4.3 · p. 6
• Many-to-many forecasting trains one function to output the full k-step sequence and requires choosing k in advance.The function maps a lag-length input window to the sequence of k forecasts.Source: §4.3 · p. 6
• Multi-neural-network forecasting trains k separate functions, one for each forecast horizon, and requires choosing k in advance.The functions are evaluated on the most recent lag-length window.Source: §4.3 · p. 6

5 RAW VS SYMBOLIC FORECASTING

• ABBA-LSTM trains on ABBA symbol strings and converts forecast strings back into numerical values using the patching procedure.The authors identify dimensional reduction, reduced parameter sensitivity, and forecasts restricted to previously seen patches as advantages of the symbolic approach.Source: §5 · p. 6

5.1 Experimental setup

• Both model types use two LSTM layers, while raw LSTM maps its final hidden state through one linear neuron and ABBA-LSTM uses k softmax output neurons.Experiments use Keras-default weight initialization, Adam, early stopping, and a fixed learning rate of 0.001; their loss functions are MSE for raw LSTM and categorical cross-entropy for ABBA-LSTM.Source: §5.1 · p. 7

5.2 Study of parameter sensitivity

• Stateful training generally produced lower DTW distances than stateless training, while raw LSTM forecasts were generally worse and more frequency-sensitive than ABBA-LSTM forecasts.The raw LSTM distance dropped near n = 40, when lag 50 covered two oscillations; ABBA-LSTM was more robust to frequency changes and remained near its default compression tolerance, tol = 0.1, k ≈ 1.41.Source: §5.2 · pp. 7–9, Figure 10 · p. 8

5.3 Need for pre-processing

• For a linear trend from 0 to 0.5, raw LSTM failed to forecast values much above 0.5, whereas ABBA-LSTM could forecast beyond the training range.ABBA uses time-series increments to capture linear trends directly; differencing raw data is an alternative, but can amplify noise.Source: §5.3 · p. 9

5.4 Shape-constraint forecasts

• ABBA-LSTM forecasts are restricted to patches seen previously, which can produce more natural-looking forecasts than unconstrained raw LSTM outputs.On HouseTwenty, which switches between [340, 370] and [2450, 2550], the raw forecasts oscillated around the range mean while ABBA-LSTM forecasts looked more like the original data.Source: §5.4 · p. 9

6.2 UCR Classification Archive

• The UCR test retained 68 z-normalized series with at least 100 raw training values and 20 ABBA symbols, using identical untuned hyperparameters for both models.Models used two 50-cell layers, 50% dropout, stateful training with lag 10, and forecasts of 50 steps.Source: §6.2 · pp. 9–10

6.1 M3 competition

• For M3 series N1500, the theta forecast was approximately straight and trend-focused, while ABBA-LSTM most closely resembled the original series among the illustrated forecasts.The paper notes that a theta forecast can achieve a small sMAPE while still appearing unrealistic to a practitioner.Source: §6.1 · p. 9, Figure 13 · p. 10

6.2 UCR Classification Archive

• In two selected UCR examples, ABBA-LSTM was visually closer to historical data in one case and forecast spiky behavior where raw LSTM was nearly constant in the other.In the Coffee example, ABBA-LSTM was farther from the truth than raw LSTM despite looking more like the historical data.Source: Figure 14 · p. 10, §6.2 · p. 10, Figure 15 · p. 11
• Across 68 UCR series, average sMAPE was 94.85 for raw LSTM versus 88.39 for ABBA-LSTM; average runtime was 1293 versus 605 seconds per series, respectively.The authors report comparable results across all four similarity measures and say ABBA-LSTM was significantly faster in most cases.Source: §6.2 · pp. 10–11, Figure 16 · p. 11, Figure 17 · p. 11

• The proposed forecasting approach combines machine learning methods for text generation with the ABBA symbolic representation.Source: §7 · p. 11
• The authors suggest that many of the ideas could extend to other recurrent networks, including Gated Recurrent Units and GPT-2.Source: §7 · p. 11
• For sufficiently long time series, the combined approach can significantly speed training without degrading forecast accuracy while reducing sensitivity to certain hyperparameters.Source: §7 · p. 11
• Future research will investigate automatically choosing the number of LSTM cells and layers based on symbolic-representation complexity.Source: §7 · p. 11