Source-linked AI summary
Long-term Forecasting with TiDE: Time-series Dense Encoder
Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, Rose Yu
TL;DR
Long-term forecasting benchmarks have shown that simple linear models can challenge Transformer-based approaches, while linear models lack support for nonlinear dependencies and covariates. TiDE uses dense MLPs to encode past values with covariates and decode forecasts with future covariates. It matches or exceeds prior neural baselines, reports over 10% lower Mean Squared Error on the largest dataset, and is 5–10x faster than the best Transformer-based model.
Problem
Simple linear models can outperform Transformers in long-term forecasting but are ill-suited to nonlinear dependencies and covariates.
Method
TiDE is a dense-MLP encoder-decoder that encodes past time-series values with covariates and decodes forecasts with future covariates.
Results
TiDE matches or exceeds prior neural baselines, with >10% lower Mean Squared Error on the largest dataset and 5-10x faster training and inference than the best Transformer-based model.
Takeaways & Limitations
Self-attention might not be necessary to learn periodicity and trend patterns on these long-term forecasting benchmarks.
Takeaways & Limitations
Reported benchmark results include a correction for a test-dataloader bug affecting PatchTST results on smaller datasets such as ETTh1 and ETTh2.
Abstract
from arXiv · showhide
Recent work has shown that simple linear models can outperform several Transformer based approaches in long term time-series forecasting. Motivated by this, we propose a Multi-layer Perceptron (MLP) based encoder-decoder model, Time-series Dense Encoder (TiDE), for long-term time-series forecasting that enjoys the simplicity and speed of linear models while also being able to handle covariates and non-linear dependencies. Theoretically, we prove that the simplest linear analogue of our model can achieve near optimal error rate for linear dynamical systems (LDS) under some assumptions. Empirically, we show that our method can match or outperform prior approaches on popular long-term time-series forecasting benchmarks while being 5-10x faster than the best Transformer based model.
1 Introduction
Long-term forecasting has exposed limits in both Transformer-based and simple linear approaches. TiDE addresses this with a simple MLP encoder-decoder, while its linear analogue has theoretical and empirical support and the full model is faster than Transformer baselines.
- Motivation: Simple linear models can outperform Transformer-based models on several long-term forecasting benchmarks, but struggle with nonlinear dependencies and covariates.The cited discussion identifies these limitations as motivation for a richer architecture.
- Architecture: TiDE is an MLP-based architecture without self-attention, recurrent, or convolutional mechanisms, giving it linear computational scaling with context and horizon lengths.Its design targets the simplicity and speed of linear models while retaining a deep learning architecture.
- Architecture: TiDE encodes past time-series values with covariates and decodes the time series together with future covariates using dense MLPs.The encoder-decoder explicitly incorporates covariates at both stages.
- Theory: A simplified linear analogue can achieve near-optimal error rates for linear dynamical systems when the LDS design matrix has maximum singular value bounded away from 1.The paper also empirically verifies this result on simulated data against LSTMs and Transformers.
- Empirical results: >10% lower Mean Squared Error is reported on the largest real-world benchmark dataset, while TiDE is 5x faster in inference and more than 10x faster in training than the best Transformer model.Across popular benchmarks, the model achieves better or similar performance to prior neural-network baselines.
2 Background and Related Work
Long-term forecasting research spans multivariate and univariate models, with many neural approaches using approximations to self-attention. Prior work also shows that simple global linear models can outperform Transformer-based multivariate approaches, motivating alternative designs and careful common-task evaluation.
- Model categories: Multivariate models jointly predict all time series using the past of interrelated variables, whereas univariate models use only one series’ history and covariates.The distinction concerns which series information enters the model during inference.
- Neural forecasting: Transformer-based forecasting models commonly use sub-quadratic approximations to full self-attention, including sparse, decomposed, frequency-enhanced, and pyramidal designs.The related approaches listed include LongTrans, Informer, Autoformer, FEDFormer, and Pyraformer.
- Univariate models: Local univariate models are typically trained separately for each time-series variable, including classical approaches such as AR and ARIMA.Different variables therefore use different models in this category.
- Univariate models: Global univariate models train one shared model across time series, and DLinear showed that a simple linear global univariate model can outperform Transformer-based multivariate approaches.DLinear learns a linear mapping from context to horizon.
- Alternative sequence models: RNN and state-space-model work has improved long-range sequence modeling, and the paper compares its method with S4 among such approaches.The cited passage places these methods in broader sequence-modeling research rather than exclusively forecasting work.
- Evaluation: All model categories can be compared fairly on multivariate long-term forecasting when evaluated on the same test set for the same task.This is the evaluation protocol followed in Section 5.
3 Problem Setting
The forecasting task predicts a multivariate horizon from historical look-back values while incorporating dynamic and static covariates. Prediction accuracy is assessed by comparing forecasts with actual future values using a metric such as MSE.
- Forecasting task: The dataset contains N time series, and the forecaster predicts each horizon y^(i)_{L+1:L+H} from its look-back y^(i)_{1:L}.The look-back length is L and the forecast horizon length is H.
- Covariates: Dynamic covariates x^(i)_t are r-dimensional features known at time t, such as weekdays, holidays, or product discounts.They may be global across series or specific to an individual time series.
- Covariates: Static attributes a^(i) describe time-invariant properties of a time series, such as product features in retail.The problem setting allows both dynamic covariates and static attributes.
- Forecasting function: A forecaster maps historical values, dynamic covariates across the look-back and horizon, and static attributes to a prediction of the future.This formalizes the inputs available for forecasting with covariates.
- Evaluation: Forecast accuracy is measured by a metric quantifying closeness between predictions and actual values, such as Mean Squared Error.MSE is presented as an example goodness-of-fit measure.
4 Model
TiDE is an MLP-based encoder-decoder for long-term forecasting that incorporates covariates and nonlinearities while retaining linear computational scaling. It projects covariates, encodes past data with static and projected features, decodes horizon-specific representations, and combines them with future covariates and residual connections.
- Motivation: TiDE addresses the limitations of linear models by adding MLP nonlinearities while modeling dependence on past data and covariates.Linear models can miss inherent nonlinearities and covariate effects; TiDE is designed to handle both.
- Encoding: Dynamic covariates are projected to a lower-dimensional space before the encoder combines them with the look-back and static attributes.Projection reduces each covariate vector from dimension r to ˜r, with ˜r ≪ r, limiting the flattened input size.
- Encoding: The encoder forms a dense representation from past values, projected past and future covariates, and static attributes using multiple residual blocks.The architecture separates feature projection from the dense MLP encoder.
- Decoding: The dense decoder maps the encoding to horizon-specific vectors, then the temporal decoder combines each vector with that time-step’s projected future covariates to produce predictions.The temporal decoder is a residual block with output size 1.
- Residual connections: The temporal decoder creates a direct highway from future covariates to each corresponding prediction, while a global linear residual maps the look-back to the horizon.The covariate highway can preserve strong direct effects, such as holiday impacts on retail demand.
- Efficiency and training: TiDE uses no self-attention, recurrence, or convolution, giving linear computational scaling with context and horizon lengths while retaining a linear model as a subclass.The model is trained with mini-batch gradient descent and mean squared error, and evaluated using rolling test-set evaluation.
5 Experimental Results
Experiments across long-term forecasting benchmarks show that TiDE is competitive with or better than strong baselines, while offering substantial efficiency advantages and benefits from its temporal decoder and residual connections.
- Benchmark results: TiDE, PatchTST, N-HiTS, and DLinear substantially outperform the other evaluated baselines across datasets.
- Benchmark results: TiDE significantly outperforms DLinear in all settings except horizon 192 on ETTh1, where their performances are equal.This supports the value of TiDE's additional non-linearity.
- Demand forecasting: Using all covariates, TiDE outperforms DeepAR by as much as 20% on the M5 competition metric.A date-derived-covariates-only version also outperforms the other baselines, although performance degrades without dataset-specific covariates.
- Efficiency: TiDE inference shows an order-of-magnitude time advantage over PatchTST, while PatchTST is more sensitive to look-back size and runs out of memory for L ≥1440.The encoder's inference scales approximately as Õ(neh^2 + hL), compared with PatchTST's approximately Õ(KnaL^2/P^2).
- Ablation studies: Removing residual connections causes a statistically significant performance drop for horizons 96–336 on Electricity.The temporal decoder also reduces post-event prediction disruption after only one epoch on a modified Electricity dataset.
6 Conclusion
The paper concludes that a simple MLP encoder-decoder can match or exceed neural forecasting baselines while substantially improving speed. Its results suggest self-attention may not be necessary for learning periodicity and trends on these benchmarks, while theory provides partial explanation under LDS assumptions.
- TiDE matches or supersedes prior neural network baselines on popular long-term forecasting benchmarks.
- TiDE is 5-10x faster than the best Transformer-based baselines.
- Self-attention might not be necessary to learn periodicity and trend patterns on these long-term forecasting benchmarks.
- Theoretical analysis proves that linear models can achieve near-optimal rates when the ground truth is generated by a linear dynamical system.
- Transformers are generally more parameter-efficient than MLPs but more memory- and compute-intensive, limiting training of extremely large-scale pretrained models.
A Theoretical Analysis under Linear Dynamical Systems
The theoretical analysis studies TiDE’s simplest linear analogue, which maps context and covariates directly to the forecasting horizon under a sufficient encoding size.
- When residual connections are active and encoding size is at least the horizon length, TiDE reduces to a linear map from context and covariates to the horizon.
A.1 Theoretical Results
The appendix formalizes forecasting from linear dynamical systems and analyzes bounded autoregressive predictors. Under stability and boundedness assumptions, a short look-back window can compete with the best LDS predictor with high probability.
- Definitions and setup: An LDS maps input vectors to output vectors through hidden states, matrices A, B, C, and D, and process and observation noise.
- Definitions and setup: Each training example contains the available sequence history and covariate information needed to predict the final observation.
- Definitions and setup: The analysis assumes bounded inputs and outputs and restricts LDS transition matrices so their eigenvalues remain below one.
- Hypothesis class: The autoregressive hypothesis class uses a concatenated look-back vector and a bounded-norm linear map to predict the target.
- Theoretical results: With k = Θ(log(1/ε)), the learned predictor is competitive with the best LDS predictor when the largest transition-matrix eigenvalue is strictly below one.
- Theoretical results: An autoregressive model with look-back length Ω(log(1/ε)) approximates an LDS with error ε, while Rademacher analysis supplies a generalization bound.
A.2 Experimental Results on Synthetic Datasets
Synthetic experiments evaluate linear, LSTM, and Transformer models on long-horizon data generated by an LDS. The linear model produces predictions closest to the truth, consistent with the reported MSE results.
- Dataset: The synthetic dataset uses a 30-dimensional LDS with operator norm 0.95, Gaussian noise and inputs, hidden cosine seasonality, and equal 320-step look-back and horizon windows.
- Baselines and setup: The comparison evaluates a direct linear history-to-future map, a one-layer LSTM, and a two-layer Transformer with self-attention.
- Results: The linear model’s predictions are closest to the ground truth, matching the MSE comparison in Table 4.
- Results: The Transformer captures lower-frequency seasonality but does not appear to leverage covariates for short-term variation, while the LSTM does not capture trend or seasonality correctly.
B.1 Additional Experiments
Additional experiments compare TiDE with S4 and detail the M5 forecasting setup, including a sparse-count loss and early stopping.
- Comparison against S4: TiDE vastly outperforms S4 on the reported time-series benchmarks.The S4 values are taken directly from Table 14 of the original S4 paper.
- M5 Forecasting: The M5 setup uses dynamic date and promotion features alongside static categorical attributes such as category, store, department, and item identifiers.The categorical features are embedded into learnable embeddings.
- M5 Forecasting: The models use a zero-inflated negative binomial likelihood and train for up to 100 epochs with early stopping patience of 5.This loss is applied to both TiDE and DeepAR for sparse count data.
B.2 Data Loader
The data-loader and experimental configuration specify look-back and forecast windows, time-derived covariates, fixed training settings, and dataset-specific hyperparameters.
- Data Loader: Each training batch contains a look-back sequence and a forecast horizon, with the batch index ranging over valid training positions.The batch size is configurable, and all series are loaded together when it exceeds the number of series.
- Experimental Configuration: The experiments report standard error bars over five independent runs and provide benchmark and hyperparameter tables for the evaluated settings.Table 6 compares TiDE with S4, while Tables 7 and 8 describe hyperparameter ranges and dataset-specific choices.
- Data Loader: Time indices from the look-back through forecast horizon are converted into eight normalized periodic covariates.Examples include minute of hour, hour of day, and day of week; some remain constant at coarser granularities.
- Experimental Configuration: Training fixes batchSize at 512 and temporalWidth at 4 while tuning architecture, normalization, dropout, and learning-rate settings.The maximum learning rate feeds a cosine-decay schedule.