PaperlayerView exact revision on arXiv

Source-linked AI summary

Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting

Xinghong Fu, Yanhong Li, Georgios Papaioannou, Yoon Kim

arXiv:2602.17634v2cs.LGcs.AI

TL;DR

Large time-series foundation models can be expensive to deploy, particularly where memory is limited. Reverso uses compact hybrid sequence mixing and a practical training and inference recipe; across benchmarks, it achieves competitive performance with substantially fewer parameters and less inference memory than much larger models.

  • Problem

    Large time-series foundation models can be expensive to deploy, especially in settings where available memory is orders of magnitude below cloud deployment.

    Source: §1 · p. 2

  • Method

    Reverso combines multi-scale inputs, hybrid convolutional and DeltaNet sequence mixing, an attention-based decoder, and a practical training and inference recipe.

    Source: §1 · p. 2, §5 · p. 4

  • Results

    Across forecasting, classification, and anomaly-detection benchmarks, Reverso achieves competitive performance with substantially fewer parameters and less inference memory than much larger TSFMs.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The results suggest compact foundation models can be a practical alternative to continued scaling in deployment-constrained settings.

    Source: §5 · p. 4

    Reverso is trained primarily as a univariate forecasting model, and its performance on shorter sequences still lags behind larger TSFMs.

    Source: §4 · p. 4

Abstract

from arXiv · show

Learning time series foundation models has been shown to be a promising approach for zero-shot time series forecasting across diverse time series domains. Insofar as scaling has been a critical driver of performance of foundation models in other modalities such as language and vision, much recent work on time series foundation modeling has focused on scaling. This has resulted in time series foundation models with hundreds of millions of parameters that are, while performant, inefficient and expensive to use in practice. This paper describes a simple recipe for learning efficient foundation models for zero-shot time series forecasting that are orders of magnitude smaller. We show that large-scale transformers are not necessary: small hybrid models that interleave long convolution and linear RNN layers (in particular DeltaNet layers) can match the performance of larger transformer-based models while being more than a hundred times smaller. We also describe several data augmentation and inference strategies that further improve performance. This recipe results in Reverso, a family of efficient time series foundation models for zero-shot forecasting that significantly push the performance-efficiency Pareto frontier.

1. Introduction

Time-series foundation models offer zero-shot forecasting across domains, but scaling them can make training and deployment expensive, especially where memory is constrained. Reverso proposes a compact hybrid architecture and practical recipe to improve the performance-efficiency frontier.

1. Introduction

  • Forecasting supports applications including weather, energy-grid analysis, supply-chain logistics, and financial prediction.Source: §1 · p. 1
  • Earlier forecasting models were often dataset-specific, requiring domain- or task-specific training and substantial data collection and model maintenance.Source: §1 · p. 1
  • Time-series foundation models enable decoder-based zero-shot forecasting from historical context, but scaling them can make training and deployment prohibitively expensive.Limited memory makes this especially challenging for on-device and edge deployment.Source: §1 · pp. 1–2
  • Reverso combines long convolutions, DeltaNet layers, multi-scale inputs, and practical data and inference strategies in models with 0.2M–2.6M parameters.The paper presents the family as a way to improve the performance-efficiency frontier for time-series foundation models.Source: §1 · p. 2

2. Methods

Reverso processes fixed-length, multi-scale inputs through alternating sequence-mixing and channel-mixing blocks, then uses an attention-based decoder to predict output patches. Its architecture combines long convolutions and DeltaNet with normalization, autoregressive inference, and a masked MAE training objective.

2.1. Architecture

  • The model maps a fixed-length input to prediction patches, using stacked sequence-mixing and channel-mixing blocks followed by an attention-based decoder.At inference, it autoregressively predicts chunks until the requested forecast length is reached.Source: §2.1 · p. 2
  • Four downsampled input channels provide an effective context of 16,384 points while keeping the model input length L fixed.The multi-scale design also encourages modeling time series at different scales.Source: §2.1 · p. 2
  • Each input channel is min-max normalized to [0, 1], missing values are linearly interpolated, and the normalized numeric sequence is projected into d dimensions.Short sequences are back-filled with their leftmost available value; outputs are unnormalized for prediction.Source: §2.1 · p. 2
  • Sequence mixing alternates gated depthwise-separable long convolutions with DeltaNet, and FFT convolution has complexity O(dL log L).The long convolution uses kernel length L; the paper notes FFT-based convolution enables faster training than standard attention.Source: §2.1 · pp. 2–3
  • DeltaNet updates a recurrent state from query, key, and value vectors, while state weaving adds the previous layer’s final time step to the current layer’s first hidden state.The update uses an input-dependent β, and the authors report that state weaving works well in practice.Source: §2.1 · p. 3
  • DeltaNet performs well despite having fewer parameters than the compared Gated DeltaNet, Gated Delta Product, and Gated Linear Attention variants.Source: §2.1 · p. 3
  • Channel mixing uses a fourfold-expansion ReLU MLP, which the authors found worked better than GLU variants; the attention-based decoder head outperformed a simple linear prediction head.Source: §2.1 · p. 3
  • Training uses mean absolute error on unnormalized predictions, masking NaN ground-truth values during loss computation.Source: §2.1 · p. 3

3. Empirical Study

The empirical study evaluates Reverso against time-series foundation-model baselines on Gift-Eval and LTSF, including forecasting quality and inference efficiency. Reverso is competitive at a modest parameter count and is faster and more memory-efficient than substantially larger models.

3.1. Experimental Setup

  • Experiments compare 200K-, 550K-, and 2.6M-parameter Reverso models with a broad set of state-of-the-art TSFMs.The baselines span varying architectures and model sizes.Source: §3.1 · p. 3

3.2. Main Results

  • 0.706 MASE: Reverso’s 2.6M-parameter model is competitive on Gift-Eval and outperforms similarly small Super-Linear, FlowState, and Tiny-Time Mixers.Figure 2 reports Gift-Eval MASE.Source: §3.2 · p. 3, Figure 2 · p. 4
  • Reverso achieves strong long-sequence point-forecasting results despite being the smallest model family evaluated on Gift-Eval.Table 2 averages MASE across 21 long-sequence datasets with all three forecast horizons available.Source: Table 2 · p. 4, §3.2 · p. 4
  • On LTSF, Reverso outperforms Sundial, Super-Linear, Timer-XL, and several other models at a much smaller parameter count.The comparison is reported for the LTSF test set; some baselines include in-domain datasets such as Electricity in pretraining.Source: §3.2 · p. 4
  • Reverso variants are faster and use less inference memory than substantially larger TSFMs while remaining competitive in forecasting.Latency and peak memory are measured on Gift-Eval for single-sample inference on one H100.Source: §3.2 · p. 4, Figure 3 · p. 4

4. Discussion

Across Gift-Eval and LTSF, Reverso matches or approaches models one to three orders of magnitude larger while improving inference latency and memory usage. The authors identify univariate training, weaker shorter-sequence performance, and a focus on point prediction as limitations.

4. Discussion

  • Across Gift-Eval and LTSF, Reverso matches or approaches models one to three orders of magnitude larger while improving inference latency and memory usage.The authors suggest that performance on current TSFM benchmark workloads also depends on how model capacity is allocated to time-series structure.Source: §4 · p. 4
  • Reverso is trained primarily for univariate forecasting, and its performance on shorter sequences still lags behind larger TSFMs.The authors identify learning cross-channel dependence and improving shorter-sequence performance as directions for further work.Source: §4 · p. 4
  • The work focuses primarily on point prediction, although some applications would benefit from distributional predictions.The authors note conformal methods as a possible lightweight route to uncertainty bounds and cite preliminary results in that direction.Source: §4 · p. 4

5. Conclusion

The conclusion presents Reverso as a parameter-efficient family of zero-shot time-series forecasting models with competitive benchmark performance and reduced parameter and inference-memory requirements. The related-work section situates it among dedicated time-series foundation models and approaches that reuse language models for time-series tasks.

5. Conclusion

  • Across forecasting, classification, and anomaly-detection benchmarks, Reverso achieves competitive performance with substantially fewer parameters and less inference memory than much larger TSFMs.The authors conclude that architecture and recipe design can shift the performance–efficiency frontier and make compact models a practical alternative in deployment-constrained settings.Source: §5 · p. 4

A. Related Works

  • Prior TSFM work includes decoder-only transformers, masked encoders, patch-based models, and approaches that adapt large language models to time-series tasks.The related-work discussion notes studies suggesting LLM backbones often provide little benefit over simpler LLM-free baselines.Source: A. Related Works · p. 9

B. Additional Methodology Details

The additional methodology details describe how Reverso balances pretraining data, augments real series, generates synthetic series, and modifies inference. They also report that flip-equivariant inference reduces forecasting error consistently across multiple benchmarks.

B. Additional Methodology Details

  • The GiftEval pretraining set contains about 4.5 million series and 230 billion time points; sampling is capped at 100,000 samples per dataset to address dataset imbalance.Random start points are chosen each epoch to sample across the full pretraining set.Source: §B.1 · p. 9
  • The augmentation pipeline applies downsampling, amplitude modulation, temporal reversal, sign inversion, censoring, and mixup in that order.Downsampling and amplitude modulation operate on full sequences; flips and censoring operate on subsampled contexts, while mixup operates on the full batch.Source: §B.1 · p. 9
  • Synthetic pretraining data combines Gaussian-process series from composite kernels and sampled means with spike processes and TSI; the recipe generates one million sequences up to length 4096.Composite kernels are formed by sampling one to five kernels and combining them with random additive or multiplicative operations.Source: §B.1 · p. 9
  • Passing both the original and sign-flipped context to the model and averaging the forecasts reduces error consistently across multiple benchmarks.This flip-equivariant strategy requires two forward passes.Source: §B.2 · p. 9
  • When FFT analysis identifies seasonality beyond the model context length, inference downsamples the input and upsamples the shorter forecast by linear interpolation.The resulting inference-time downsampling is applied in addition to the downsampling used to create multi-scale inputs.Source: §B.2 · p. 10

C.1. Classification Benchmarks

These benchmarks assess Reverso as a frozen feature extractor and as a zero-shot forecaster, anomaly detector, imputer, and probabilistic forecaster. Results are competitive across several settings, with reported strengths in anomaly detection and imputation and stable Gift-Eval performance across seeds.

C.1. Classification Benchmarks

  • Reverso matches the best-performing baselines on UCR and UEA without task-specific fine-tuning; on the full 128-task UCR archive, it obtains 81% accuracy.The full-archive result is reported separately from the filtered comparison protocol.Source: §C.1 · p. 11, Table 3 · p. 11

C.2. LTSF/TSLib

  • Reverso outperforms Sundial, Super-Linear, Timer-XL, and several other models on LTSF at a much smaller parameter count.Figure 5 compares MAE averaged over four horizons across six datasets against parameter count.Source: §C.2 · p. 11, Figure 5 · p. 12

C.3. Anomaly Detection

  • Reverso detects strictly more anomalies than baselines at every threshold, including q = 1%, where most baseline forecasters miss the labeled anomaly.The benchmark reports detected counts at q ∈ {1%, 3%, 10%}.Source: §C.3 · p. 12, Table 5 · p. 12

C.4. Imputation

  • Reverso achieves the lowest imputation MSE on all seven standard benchmarks.The evaluation fine-tunes the pretrained model end-to-end in a masked-reconstruction setting.Source: §C.4 · p. 12, Table 6 · p. 13

C.5. Probabilistic Forecasting

  • Reverso remains competitive in probabilistic forecasting and obtains the best CRPS score on the medium-horizon Gift-Eval split.This experiment adapts a pretrained backbone by training its decoder head with quantile regression rather than training the full model from scratch for probabilistic forecasting.Source: §C.5 · p. 13, Table 7 · p. 13

C.6. Robustness Across Seeds

  • Across four completed runs, Reverso’s final Gift-Eval MASE has a mean of 0.7115 and standard deviation of 0.0013.The paper describes this result as stable across random seeds.Source: §C.6 · p. 13, Table 8 · p. 13

D.1. Synthetic data composition

The synthetic-data and augmentation recipe combines varied generated time series with sequential instance transformations and batch-level Mixup. Its described components target diverse temporal patterns and resolutions.

D.1. Synthetic data composition

  • KernelSynth composes sampled kernels with random addition or multiplication, then samples a synthetic series from a Gaussian process with a constant or linear-trend mean.The kernel bank includes periods spanning multiple timescales.Source: §D.1 · pp. 13–14
  • The synthetic mix also includes spike processes and TSI generation to help the model learn simple trends and periodic patterns.The spike generator includes periodic trapezoid patterns, while TSI generation includes trend, seasonality, noise, outliers, and level shifts.Source: §D.1 · pp. 13–15

D.2. Data augmentation specifics

  • Downsampling compresses long-term dependencies into the context window, while amplitude modulation multiplies each series by a piecewise-linear function with one intermediate changepoint.These are sequential instance-level augmentations in the described pipeline.Source: §D.2 · p. 16
  • The pipeline also applies sign and temporal flips, clips values from both tails to reduce anomaly effects during training, and mixes samples by convex interpolation at batch level.The augmentation order is downsampling, amplitude modulation, flips, censoring, then Mixup.Source: §D.2 · p. 16

E. Extended results on Gift-Eval

The supplied figures and tables describe Gift-Eval results across sequence lengths, forecasting horizons, and domains. Their captions identify evaluation scope and visualizations but do not provide additional benchmark outcomes beyond the stated overall MASE.

E. Extended results on Gift-Eval

  • Figure 6 separates Gift-Eval zero-shot performance into long-sequence tasks, with average length at least 2048, and short-sequence tasks.The caption identifies the two panels but does not state which performs better.Source: Figure 6 · p. 18
  • Reverso’s full Gift-Eval results in Table 10 have an overall MASE of 0.706.The caption does not provide a breakdown by dataset or horizon.Source: Table 10 · p. 19
  • Table 11 presents MASE scores alongside parameter sizes; its data-leaked Chronos-2 version is excluded from Figure 2’s fair zero-shot comparison.The caption describes the leaked version as a strong reference for state-of-the-art foundation models.Source: Table 11 · p. 20
  • Figure 7 visualizes forecasts across Gift-Eval tasks, with the caption describing long-horizon forecasts that capture patterns at multiple frequency scales.A vertical dotted line marks the length of one autoregressive prediction.Source: Figure 7 · p. 21

F. Detailed results for LTSF/TSLib

The LTSF results table summarizes MAE across six datasets and four horizons, with missing model scores imputed from the best available model. The downsampling procedure tests for significant seasonality before expanding the context’s effective temporal reach.

F. Detailed results for LTSF/TSLib

  • Table 13 reports results across six datasets averaged over horizons {96, 192, 336, 720}, with missing values imputed from the best available model for each horizon.The table covers models shown in Figure 5.Source: Table 13 · p. 23

G. Downsampling Algorithm

  • A seasonality peak at f1 is significant only when its amplitude p1 is at least α times the second-highest peak p2, the DC amplitude pDC, and mean spectral amplitude plus β standard deviations.This single-dominant-frequency test mitigates ambiguity from multi-scale seasonality, which is left for future work.Source: G. Downsampling Algorithm · p. 24
  • Taking every k-th point expands the effective receptive field to k · L while preserving input length L, but downsampling is omitted for short-term forecasts when it would reduce prediction resolution.The paper averages downsampling ratios for the same frequency within a dataset because sample-level ratios vary substantially; its period-4000 example illustrates how downsampling can fit multiple periods in context.Source: G. Downsampling Algorithm · pp. 24–25, Figure 8 · p. 24

H. Ablations

The ablations find benefits from multiscale inputs, hybrid convolution–DeltaNet mixing, and an attention decoder, while data augmentations and layer ordering also affect the recipe.

Ablations

  • Ablations use a smaller portion of the training set because of compute constraints; differences above the reported 0.002–0.003 seed variation are treated as significant.Source: H. Ablations · p. 25

Architecture

  • Multiscale downsampling by up to 8× improves Gift-Eval long-horizon MASE across Reverso-Nano, Reverso-Small, and Reverso.The strategy uses C = 4 frequency channels and aims to capture multiple frequency scales and extend effective context.Source: H. Ablations · p. 25, Table 14 · p. 25
  • Hybrid long-convolution and linear-RNN models perform best; linear attention and convolution methods consistently outperform full attention.Across non-hybrid models, DeltaNet and Gated DeltaNet achieve low loss with few parameters compared with Gated Linear Attention and DeltaProduct.Source: H. Ablations · p. 25, Table 15 · p. 25
  • The attention decoder boosts overall performance, particularly for capturing long-range dependencies.The ablation replaces the attention mechanism with a simple (bi)linear layer.Source: Table 15 · p. 25, H. Ablations · p. 26

Data pipeline ablations

  • Removing any single data augmentation does not significantly hurt pre-training, but removing augmentations altogether is detrimental, and synthetic data benefits performance even at small ratios.Source: Table 16 · p. 26

Hybrid layer ordering

I. Mechanistic analysis

The mechanistic analysis finds convolution filters with temporal and frequency selectivity, while DeltaNet states respond to periodic signals and gradual temporal evolution. The authors posit that this division helps explain the hybrid strategy’s effectiveness.

Convolution filters

  • Convolution filters include weights biased toward recent temporal positions and low-pass filters spanning a few hundred time steps.Source: §I · p. 27, Figure 9 · p. 27
  • Later Reverso convolution layers act as higher-frequency filters, and the authors believe alternating layers let DeltaNet blocks operate across varying frequency scales.The frequency analysis covers convolution layers across Reverso (2.6M).Source: §I · p. 27, Table 19 · p. 27

DeltaNet states

  • Visualized DeltaNet states show sensitivity to periodic data, with some state portions tracking gradual temporal evolution.The example states show both high-frequency variations and low-frequency graduations in response to periodic signals.Source: §I · p. 27, Figure 10 · p. 27

Interpretation

  • The authors posit that the hybrid strategy’s effectiveness is driven by convolutions learning position-specific information and DeltaNet learning periodic information.Source: §I · p. 27

Related papers

Loading 2602.17634v2…