PaperlayerView exact revision on arXiv

Source-linked AI summary

TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning

Andreas Auer, Patrick Podest, Daniel Klotz, Sebastian Böck, Günter Klambauer, Sepp Hochreiter

arXiv:2505.23719v2cs.LG

TL;DR

Zero-shot time series forecasting needs models that combine in-context learning with the state-tracking useful for long horizons. TiRex uses xLSTM, Contiguous Patch Masking, and data augmentations to address this gap, and establishes state-of-the-art performance across short- and long-term forecasts on GiftEval and Chronos-ZS. Its main scope is univariate forecasting, with broader multivariate modeling left for future work.

  • Problem

    Transformers commonly support in-context learning but can underperform in time series forecasting, while LSTMs provide state-tracking but lack strong in-context learning capabilities.

  • Method

    TiRex combines an xLSTM backbone with Contiguous Patch Masking and three data augmentations for zero-shot time series forecasting.

  • Results

    TiRex establishes a new state of the art in zero-shot forecasting, outperforming prior methods across short- and long-term horizons on the Chronos-ZS and GiftEval benchmarks.

  • Takeaways & Limitations

    TiRex indicates that an xLSTM-based pretrained model can combine state-tracking with strong zero-shot forecasting across short and long horizons.

  • Takeaways & Limitations

    TiRex focuses on univariate time series; incorporating multivariate data remains future work.

Abstract

from arXiv · show

In-context learning, the ability of large language models to perform tasks using only examples provided in the prompt, has recently been adapted for time series forecasting. This paradigm enables zero-shot prediction, where past values serve as context for forecasting future values, making powerful forecasting tools accessible to non-experts and increasing the performance when training data are scarce. Most existing zero-shot forecasting approaches rely on transformer architectures, which, despite their success in language, often fall short of expectations in time series forecasting, where recurrent models like LSTMs frequently have the edge. Conversely, while LSTMs are well-suited for time series modeling due to their state-tracking capabilities, they lack strong in-context learning abilities. We introduce TiRex that closes this gap by leveraging xLSTM, an enhanced LSTM with competitive in-context learning skills. Unlike transformers, state-space models, or parallelizable RNNs such as RWKV, TiRex retains state-tracking, a critical property for long-horizon forecasting. To further facilitate its state-tracking ability, we propose a training-time masking strategy called CPM. TiRex sets a new state of the art in zero-shot time series forecasting on the HuggingFace benchmarks GiftEval and Chronos-ZS, outperforming significantly larger models including TabPFN-TS (Prior Labs), Chronos Bolt (Amazon), TimesFM (Google), and Moirai (Salesforce) across both short- and long-term forecasts.

1 Introduction

TiRex addresses a tension in zero-shot time series forecasting: transformers support in-context learning, while LSTMs offer effective state-tracking but weaker in-context learning. It combines xLSTM with Contiguous Patch Masking and data augmentations to improve forecasting across short and long horizons.

  • Motivation: The paper motivates zero-shot forecasting as broadly useful in domains such as energy, retail, and healthcare.Its formal setup predicts a future horizon from past observations and can model the conditional distribution over future outcomes.
  • Motivation: Transformers provide strong in-context learning, whereas LSTMs often perform strongly in time series forecasting because of recurrence and state-tracking.The paper contrasts these strengths as a central architectural gap.
  • TiRex: xLSTM is adopted to combine LSTM-based state-tracking with competitive in-context learning capabilities.The paper describes xLSTM as a modern LSTM variant with architectural enhancements for scalability and generalization.
  • TiRex: Contiguous Patch Masking enhances state-tracking and supports coherent long-horizon predictions by mitigating degradation in autoregressive multi-step forecasting.The strategy is applied during training and is designed to facilitate reliable uncertainty estimates over long horizons.
  • Contributions: TiRex sets a new state of the art in zero-shot forecasting across standardized benchmarks, improving both short- and long-term forecasting accuracy.The paper also introduces three augmentation techniques intended to improve robustness and overall performance.
  • Motivation: Zero-shot forecasting applies pretrained models directly to unseen datasets without parameter updates or task-specific supervision.This is especially useful for practitioners without machine learning expertise and in data-scarce settings.

2 TiRex

TiRex uses a decoder-only xLSTM architecture with residual input and output layers, patch-based processing, and quantile predictions. Its multi-patch inference treats future inputs as missing values so internal memory propagates predictive state and uncertainty across forecast patches.

  • Architecture: TiRex stacks multiple xLSTM blocks between lightweight input and output layers in a decoder-only architecture.The input layer scales and patches the series into tokens, while output tokens are mapped back to forecast patches.
  • Architecture: The xLSTM backbone uses sLSTM sequence mixing, which supports state-tracking, followed by feed-forward processing within each block.The blocks are preceded by RMSNorm and can be stacked according to model size.
  • Input and output: TiRex applies instance-wise z-score normalization and non-overlapping window patching before processing the series with xLSTM blocks.A binary presence mask is concatenated to values to represent missing inputs.
  • Input and output: The output layer predicts |Q| quantile values for each time step rather than single-point forecasts.The losses of output tokens are averaged, while missing values in the output window are ignored.
  • Multi-Patch Horizon Forecasts: For horizons longer than one output patch, TiRex represents future inputs as missing values, allowing internal memory to propagate predictive state and uncertainty across patches.This differs from autoregressive approaches that feed point estimates into subsequent patches and reinitialize probabilistic forecasts at each step.
  • Contiguous Patch Masking: Contiguous Patch Masking randomly masks full consecutive patches during pre-training to mirror the missing-value structure used for multi-patch inference.The procedure samples a consecutive-patch span and expands a binary mask across the corresponding time-series values.

3 Data Augmentation

TiRex uses three pre-training augmentations to expose the model to more diverse time-series dynamics and improve robustness. They modify scale, censor values, or inject transient periodic spikes.

  • Augmentations: Amplitude Modulation introduces trends and change points in a series’ scale by multiplying values by a varying amplitude.The amplitude follows a linear trend that may include change points.
  • Augmentations: Censor Augmentation clips series values at a randomly sampled threshold derived from an empirical signal quantile.The threshold is obtained by uniformly sampling a quantile from the empirical distribution.
  • Purpose: The augmentation suite is designed to expose TiRex to a wider range of potentially relevant dynamics and enhance pre-training robustness.The paper presents these techniques as an underused opportunity in time-series pre-training compared with vision pre-training.
  • Augmentations: Spike Injection adds short periodic spike signals using kernels such as tophat, radial basis function, or linear kernels.Periodicity and kernel parameters are randomized to encourage learning a general concept of transient events.

4 Experiments

Experiments evaluate TiRex on standardized zero-shot benchmarks, compare it with state-of-the-art models, and analyze efficiency and component contributions. TiRex consistently performs strongly across forecasting horizons while using fewer resources, and ablations support the roles of CPM, augmentations, and the xLSTM design.

  • Evaluation Setup: TiRex is evaluated on Chronos-ZS and GiftEval, covering diverse datasets, frequencies, forecast horizons, and 97 GiftEval evaluation settings.The Chronos-ZS benchmark contains 27 datasets, while GiftEval contains 24 datasets evaluated in different settings.
  • Zero-Shot Forecasting: GiftEval-ZS shows TiRex consistently outperforming competing methods across both short- and long-term forecasting tasks.The benchmark reports aggregated performance and average CRPS rank, with lower values indicating better performance.
  • Zero-Shot Forecasting: 0.411 CRPS is TiRex’s GiftEval-ZS score, with ±0.002 standard deviation across six training seeds, ahead of next-best zero-shot scores of 0.459, 0.463, and 0.481.TiRex is also reported as the only model excelling simultaneously at short- and long-term forecasting and surpassing PatchTST and TFT on long-term forecasts.
  • Zero-Shot Forecasting: On Chronos-ZS, TiRex achieves the best WQL score and rank, while placing second in MASE behind TabPFN-TS.Moirai’s stronger Chronos-ZS performance is accompanied by 82% overlap between its pre-training data and the test set.
  • Inference Efficiency: TiRex requires significantly less GPU memory and achieves faster inference than larger next-best models under tested context, prediction, and batch-size settings.The comparison uses context length 2048 and prediction length 32 across multiple batch sizes.
  • Ablations: Ablations indicate that CPM is essential for multi-patch prediction, augmentations improve generalization, and sLSTM-only TiRex performs especially well on long-term forecasts.Removing any augmentation decreases at least one benchmark metric, with the largest decline occurring without augmentations; mLSTM-only performs worst.

5 Conclusion

TiRex combines an xLSTM backbone with Contiguous Patch Masking to achieve strong zero-shot forecasting across short- and long-term horizons. Its main scope limitation is a focus on univariate series, with limited hyperparameter tuning.

  • 5 Conclusion: TiRex uses xLSTM and Contiguous Patch Masking to establish state-of-the-art zero-shot forecasting across Chronos-ZS and GiftEval.The model maintains strong performance across both short- and long-term horizons.
  • 5 Conclusion: Contiguous Patch Masking supports long-term forecasting without sacrificing short-term capabilities by improving xLSTM state tracking.It is designed to mitigate degradation in autoregressive multi-step forecasting.
  • 5 Conclusion: TiRex focuses on univariate time series, while future work could incorporate multivariate data through extended contexts or modified input layers.The authors also report that computational constraints limited extensive hyperparameter tuning.
  • A xLSTM: xLSTM extends LSTM architectures with enhanced scalability, parallelization, and in-context modeling capabilities.Its sLSTM pathway preserves recurrent state tracking, while xLSTM blocks combine recurrent layers with feed-forward, residual, and normalization components.

B Data Augmentation

TiRex uses several time-series augmentations during pre-training to expose the model to diverse trends, censoring patterns, and transient events. These augmentations are implemented through amplitude modulation, censoring, and structured spike injection.

  • B Data Augmentation: Amplitude Modulation introduces scale trends and change points by multiplying the signal with a piecewise linear trend.The trend is generated by sampling change points and interpolating amplitudes between them.
  • B Data Augmentation: Censor Augmentation clips the input signal from above or below using a threshold sampled from the signal’s empirical quantile distribution.The censoring direction is randomly sampled.
  • B Data Augmentation: Spike Injection adds sparse periodic spikes generated from kernels with randomized shapes and magnitudes to improve generalization to sharp transient events.Periodic patterns, kernel types, and kernel parameters are sampled to diversify spike structures.

C.1 Model and Training Hyperparameter

TiRex is a 35-million-parameter xLSTM model trained with a long context and a specified AdamW optimization schedule.

  • C.1 Model and Training Hyperparameter: TiRex has 35 million model parameters and uses an xLSTM-based architecture.Its architecture hyperparameters are summarized in Table 4.
  • C.1 Model and Training Hyperparameter: TiRex is pre-trained for 500,000 steps with batch size 256, AdamW, learning rate 0.001, weight decay 0.01, and context length 2048.Training uses cosine decay with 5% linear warm-up and minimum learning rate 0.0001.

C.2 Pre-Training Data Corpus

TiRex is trained on a diverse corpus combining real and synthetic time series. The corpus expands Chronos data with mixup and Gaussian-process-generated sequences, alongside GiftEval pre-training data.

  • C.2 Pre-Training Data Corpus: The training corpus combines real and synthetic time series to support generalization across heterogeneous forecasting tasks.It contains three dataset components.
  • C.2 Pre-Training Data Corpus: Chronos training data contributes 30 million time series, with more and longer sequences generated than in the original Chronos procedure.TiRex also adopts TsMixup for the Chronos training data.
  • C.2 Pre-Training Data Corpus: TsMixup forms convex combinations of k z-score-normalized time-series segments using Dirichlet-distributed mixing weights.Sampling k = 1 preserves original sequences in the augmented dataset.
  • C.2 Pre-Training Data Corpus: Synthetic Gaussian-process data uses composite kernels formed by randomly combining sampled base kernels with addition or multiplication.The procedure uses up to four base kernels and synthetic sequence length 4096.

C.3 Benchmarks and Metrics

TiRex is evaluated on standardized GiftEval and Chronos-ZS benchmarks spanning diverse datasets and forecast horizons, using normalized point and probabilistic metrics. Across extended results, TiRex consistently leads the zero-shot comparisons, including short-, medium-, and long-term settings.

  • GiftEval: GiftEval covers 23 datasets, seven domains, ten sampling frequencies, and 97 evaluation settings across short- to long-term horizons.
  • GiftEval-ZS: GiftEval-ZS excludes 16 settings overlapping TiRex’s pre-training data to preserve zero-shot comparability and avoid leakage.
  • Metrics: MASE evaluates point forecasts, while CRPS and WQL evaluate probabilistic forecasts using benchmark-specific quantile-based procedures.
  • Chronos-ZS: Chronos-ZS contains 27 datasets focused primarily on short-term forecasting, with no overlap between its data and TiRex’s pre-training corpus.
  • Results: TiRex outperforms all full-GiftEval baseline models by a substantial margin, with the largest gap in long-term forecasting tasks.
  • Results: 39 TiRex consistently achieves the best performance across GiftEval-ZS short-, medium-, and long-term settings and Chronos-ZS comparisons.

D.3 Multivariate data

Although TiRex models each variate independently, it remains effective on multivariate forecasting tasks. On the multivariate GiftEval-ZS subset, it ranks above models designed specifically for multivariate forecasting.

  • Multivariate performance: TiRex models each time-series variate independently yet outperforms several multivariate models on the full GiftEval-ZS benchmark.
  • Multivariate performance: On the multivariate GiftEval-ZS subset, TiRex achieves the top rank among models designed specifically for multivariate forecasting, including Moirai and TTM.
  • Evaluation view: Chronos-ZS results report aggregated scores and average WQL rank, with lower values indicating better performance.
  • Efficiency: Inference efficiency is compared using GPU memory and inference time per sample across batch sizes on an Nvidia A40 with 48 GB available memory.
  • Evaluation view: The multivariate subset reports aggregated scores and average CRPS rank for GiftEval-ZS short-term results.

D.4 Ablations

The ablations examine Contiguous Patch Masking, augmentation probabilities, architecture controls, qualitative forecasts, and fine-tuning. Results indicate that CPM is important for long-horizon forecasting, while performance is comparatively robust to several hyperparameter changes.

  • Contiguous Patch Masking: CPM uses a maximum masking probability pmax_mask and a maximum consecutive-patch count cmax_mask as its two hyperparameters.
  • Contiguous Patch Masking: Long-term forecasts require CPM and sufficient multi-patch training examples, with cmax_mask > 3 or pmax_mask ≥0.5 providing that condition.
  • Augmentations: Performance is relatively robust to augmentation application probabilities when augmentations are used, although excessive application can diminish returns.
  • Architecture control: A Chronos-Bolt architecture was trained with TiRex’s data and augmentation pipeline to isolate the contribution of that pipeline from other methodological choices.
  • Fine-tuning: Fine-tuning TiRex produces incremental improvement over the pre-trained model, especially on MASE.

E TiRex 1.1 - Full GiftEval Zero-Shot

TiRex 1.1 revises the training corpus and preprocessing to support a strictly zero-shot full-GiftEval evaluation. It maintains state-of-the-art performance across the reported metrics, while deployment remains important to consider in high-stakes settings.

  • Zero-shot data: TiRex 1.1 removes GiftEval datasets and overlapping Chronos-ZS datasets from pre-training to eliminate potential data leakage.
  • Preprocessing: Long-period normalization rescales series so a dominant period fits within TiRex’s context window, addressing long-range periodicities.
  • Full GiftEval results: The full GiftEval figures report overall, short-term, and long-term aggregated scores together with average CRPS rank, where lower values are better.
  • Full GiftEval results: TiRex 1.1 achieves the top rank across all reported metrics in the updated strictly zero-shot full-GiftEval comparison.
  • Reproducibility: TiRex’s code repository is hosted on GitHub.
  • Scope and caveat: The benchmark scores use leaderboard computations from the submission time; a later seasonal-naive update changes absolute scores but not rankings or conclusions.
Loading 2505.23719v2…