Long-term Forecasting using Higher Order Tensor RNNs

• Nonlinear multivariate systems pose a long-term forecasting challenge because they combine higher-order correlations, long temporal dependencies, and sensitivity to error propagation.Source: §1 · p. 1
• HOT-RNN models non-Markovian dynamics and higher-order interactions among previous hidden states, and uses tensor-train decomposition to make learning tractable.Source: §1 · p. 2
• The paper provides theoretical guarantees for HOT-RNN expressiveness for nonlinear dynamics and characterizes the target dynamics and their HOT-RNN representation.Source: §1 · p. 2
• HOT-RNNs forecast more accurately over significantly longer horizons than standard RNNs and LSTMs on simulated data and real-world nonlinear environments.Source: §1 · p. 2

• ARMA and ARIMA model stochastic processes under linear-dynamics assumptions, while system identification often requires strong parametric assumptions.Source: §2 · p. 2
• Many neural forecasting models study short-term prediction and use only the most recent state, unlike HOT-RNN's explicit modeling of higher-order dynamics for long-term dependencies.Source: §2 · p. 3
• Earlier higher-order RNNs include models limited to two-way interactions, input-specific hidden weights, or linear interactions among concatenated past states (Soltani and Jiang, 2016).Source: §2 · p. 3
• Tensor-train compression has been applied to neural-network weights and input-hidden RNN tensors, whereas HOT-RNN learns dynamics in hidden-state space (Novikov et al., 2015; Yang et al., 2017).Source: §2 · p. 3
• The paper presents its approach as the first use of tensor networks within RNNs for sequence learning in nonlinear-dynamics environments.Source: §2 · p. 3

3. Higher-Order Tensor RNNs

• Nonlinear dynamics can produce higher-order correlations, long-term dependencies, and sensitivity to error propagation, making accurate long-horizon forecasting challenging.The forecasting task is to predict future states from an initial sequence of states.Source: §3 · p. 4

3.1 Higher-Order Non-Markovian Models

• HOT-RNN uses L steps of temporal memory and polynomial interactions between hidden states to model higher-order dynamics.In the seq2seq setup, the encoder processes the observed states and the decoder predicts future states using its previous prediction as input.Source: §3.1 · pp. 5–6

3.2 Dimension Reduction with Tensor-Train

• Tensor-train decomposition reduces HOT-RNN parameters from (HL + 1)^P to (HL + 1)R^2P, with R bounding the tensor-train rank.Without this decomposition, the parameter count grows exponentially as O(H^(LP)), making the higher-order model prohibitively large to train.Source: §3.2 · p. 6

4. Approximation Theorem for HOT-RNNs

• The analysis uses compact tensor-train representations for functions satisfying regularity conditions, then relates HOT-RNN approximation to a functional tensor-train representation.The proof sketch assumes the target state-transition function is a Sobolev function whose weak derivatives are preserved.Source: §4 · pp. 7–8
• The approximation characterization depends on target-function regularity, input dimension, tensor-train rank, and polynomial order.The theorem considers a Hӧlder-continuous target with bounded derivatives and finite Fourier magnitude distribution.Source: §4 · p. 7
• HOT-RNNs are exponentially more powerful than first-order regular RNNs for large rank, and increasing polynomial order can reduce the required hidden units.The number of weights required also depends on the target function's regularity.Source: §4 · p. 7

Variance bound

• For dynamics of order at most P, Theorem 2 upper-bounds the variance between the HOT-RNN estimator and the population statistic.The setup assumes the dynamics are represented by a joint probability distribution; the bound depends on the sample count m and ρ.Source: §5 · p. 8

Proof strategy

• The variance analysis uses the connection between tensor-train distributions, Gibbs fields, and conditional random fields.The proof invokes the Hammersley–Clifford theorem to establish the graphical-model connection.Source: §5 · pp. 8–9

Estimator

• HOT-RNN represents P-th order dynamics as a joint distribution and estimates its feature-mapping statistic by factorizing tensor products of basis functions.The estimator uses m observed time-series samples.Source: §5 · pp. 8, 10

6.1 Datasets

• The experiments cover Genz dynamics, traffic, and climate; Dickey–Fuller tests indicate the traffic and climate series are nonstationary.The datasets comprise synthetic dynamics and real-world time series.Source: §6.1 · pp. 10–11, Figure 5 · p. 11, Table 1 · p. 11

6.2 Training Details

• Training uses seq2seq HOT-LSTM, an 80/10/10 train-validation-test split, and validation-loss early stopping; scheduled sampling was omitted because it destabilized training.Models were trained for at most 10^4 steps.Source: §6.2 · p. 11

6.3 Long-term Forecasting Accuracy

• HOT-LSTM notably outperforms all baselines across Genz, traffic, and climate datasets and is more robust to long-term error propagation.Figure 6 reports test RMSE across forecasting horizons, averaged over three runs.Source: Figure 6 · p. 12, §6.3 · p. 12

6.2 Training Details

• HOT-RNN with LSTM cells performs best in the reported experiments, while HOT-RNN with RNN cells outperforms vanilla RNN and MRNN.ARIMA consistently performs approximately 5% worse than LSTM.Source: §6.2 · p. 12

6.3 Long-term Forecasting Accuracy

• On Moving MNIST, HOT-LSTM gains approximately 2–5% over baselines when forecasting 20 and 40 frames from 10 initial frames.The authors suggest the gain is small because the circular dynamics are simple and spatial structure is difficult for RNNs without convolutional features.Source: Table 3 · p. 12, §6.3 · p. 13

6.4 Visualization of Predictions

• For Genz cornerpeak dynamics, HOT-LSTM nearly recovers the transition function, while LSTM and MLSTM predict only the mean and miss the dynamics’ range and phase.On traffic and climate data, HOT-LSTM aligns better with long-term trends and captures more curvature than MLSTM.Source: §6.4 · p. 14

6.5 Model Capacity

• The best HOT-LSTM models have parameter counts comparable to MLSTM and LSTM, while tensor-train memory complexity grows quadratically with rank as expressiveness improves exponentially.The best real-world climate and traffic model sizes are listed in Table 4.Source: §6.5 · p. 14, Table 4 · p. 14

6.6 Speed Performance Trade-off

• HOT-RNNs converge faster and reach lower validation loss than other models in the long-term forecasting evaluation.Figure 9 plots validation loss against training steps for each model’s best long-term forecasting configuration.Source: Figure 9 · p. 14, §6.6 · p. 15

6.7 Sensitivity Analysis

• Too-low tensor-train rank limits nonlinear-dynamics capacity, whereas too-high rank can overfit; the best lag varies with forecasting horizon.The sensitivity analysis reports prediction RMSE by rank at lag 3 and examines varying lags, selecting the best rank by cross-validation for each setting.Source: Table 5 · p. 14, Table 6 · p. 14, §6.7 · p. 15

6.8 Chaotic Nonlinear Dynamics

• On 45-step-ahead Lorenz predictions, tensor models outperform vanilla RNN and MRNN, while HOT-RNN shows only slight improvement at the beginning state.The figure shows HOT-RNN predictions as more consistent but imperfect, while baselines are highly unstable and noisy.Source: §6.8 · p. 15, Figure 11 · p. 16
• HOT-RNN predicts Lorenz dynamics up to T = 40 steps, but diverges quickly beyond that; no state-of-the-art model tested was stable beyond 40 steps.Chaotic dynamics make long-term forecasting difficult because small errors can lead to catastrophic errors.Source: §6.8 · p. 16

7. Discussion

• HOT-RNN directly learns nonlinear dynamics using higher-order structures and is designed for long-term forecasting.The discussion reports forecasting benefits on synthetic and real-world multivariate time series.Source: §7 · p. 16
• The paper provides approximation guarantees for HOT-RNN’s representation power.The discussion characterizes these as the first such guarantees for this model class.Source: §7 · p. 16
• Forecasting chaotic dynamics remains a significant challenge for sequential prediction models, motivating future work on robust models.The authors also propose studying HOT-RNN beyond forecasting, including sequence domains such as language.Source: §7 · p. 16

.1 Theoretical Analysis

• The proof uses tensor-train decomposition to obtain a compact representation that preserves weak differentiability for functions satisfying regularity conditions.The resulting approximation guarantee depends on target-function regularity, input dimension, and tensor-train rank.Source: .1 Theoretical Analysis · p. 20
• The analysis assumes the target state-transition function is a Sobolev function with bounded derivatives and integrability on its input space.The stated target function takes concatenated previous states as input.Source: .1 Theoretical Analysis · p. 20
• Truncating the functional tensor-train expansion projects the target function onto a finite-basis subspace, yielding an approximation error bound.The analysis then connects this tensor-train approximation to one-hidden-layer neural-network approximation.Source: .1 Theoretical Analysis · pp. 20–21
• As tensor-train rank and polynomial order increase, the required number of hidden units decreases, up to a constant depending on the regularity of the dynamics.HOT-RNN implements a polynomial expansion of input states to approximate the tensor-train representation.Source: .1 Theoretical Analysis · pp. 21–22

.2 Additional Experiments

• For oscillatory, product-peak, and Gaussian Genz functions, HOT-LSTM better captures complex dynamics, leading to more accurate predictions.Figure 14 places transition functions, dynamics realizations, and LSTM versus HOT-LSTM predictions in separate columns.Source: .2 Additional Experiments · p. 22, Figure 13 · p. 23
• Moving MNIST generates about 50,000 video sequences of length 100, with digits moving across a 48×48 canvas along trajectories using cosine and sine velocity.Figure 14 shows ground truth in the top row, LSTM predictions in the middle, and HOT-LSTM predictions in the bottom row.Source: .2 Additional Experiments · p. 22, Figure 14 · p. 24
• Figure 14 compares HOT-LSTM with baselines on visualizations of Genz functions, their dynamics, and model predictions.The figure’s columns show transition functions, dynamics realizations, and model predictions.Source: Figure 13 · p. 23