PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting

Amin Shabani, Amir Abdi, Lili Meng, Tristan Sylvain

arXiv:2206.04038v4cs.LG

TL;DR

Time-series transformers often lack explicit encouragement to learn cross-scale relationships. Scaleformer iteratively refines forecasts at multiple scales with shared weights and cross-scale normalization. The paper reports MSE reductions of 5.5% to 38.5% across transformer architectures.

  • Problem

    Transformer architectures often learn essential cross-scale relationships implicitly rather than through explicit scale-awareness.

    Source: §1 · p. 1

  • Method

    Scaleformer iteratively applies the same neural module across temporal scales and introduces normalization to reduce distribution shifts between scales.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    5.5% to 38.5% mean squared error reductions are reported across transformer architectures for the multi-scale framework.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach also improves probabilistic forecasting and shows statistically significant average improvement when adapted by NHits and FiLM.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

The performance of time series forecasting has recently been greatly improved by the introduction of transformers. In this paper, we propose a general multi-scale framework that can be applied to the state-of-the-art transformer-based time series forecasting models (FEDformer, Autoformer, etc.). By iteratively refining a forecasted time series at multiple scales with shared weights, introducing architecture adaptations, and a specially-designed normalization scheme, we are able to achieve significant performance improvements, from 5.5% to 38.5% across datasets and transformer architectures, with minimal additional computational overhead. Via detailed ablation studies, we demonstrate the effectiveness of each of our contributions across the architecture and methodology. Furthermore, our experiments on various public datasets demonstrate that the proposed improvements outperform their corresponding baseline counterparts. Our code is publicly available in https://github.com/BorealisAI/scaleformer.

1 INTRODUCTION

The introduction argues that transformers need stronger scale awareness because cross-scale relationships are often learned implicitly. Scaleformer uses iterative multi-scale refinement and cross-scale normalization, and reports MSE reductions across transformer backbones.

1 INTRODUCTION

  • Transformers often learn essential cross-scale relationships implicitly, while prior scale-aware designs focus on only low- and high-frequency components.The introduction asks whether transformers can be made more scale-aware.Source: §1 · p. 1
  • Scaleformer iteratively refines forecasts at successive time steps to capture inter-dependencies and specificities at different scales.Figure 1 presents intermediate forecasts at different time scales as an illustration of the structural prior.Source: §1 · p. 1, Figure 1 · p. 1
  • Cross-scale normalization is introduced to mitigate distribution shifts between intermediate forecasts that can cause runaway error propagation.The normalization is applied at each step.Source: §1 · p. 2
  • The approach re-orders model capacity toward scale awareness without changing transformers’ attention-driven paradigm, allowing adaptation across recent architectures.The paper tests it with FEDformer, Autoformer, Informer, Reformer, and Performer.Source: §1 · p. 2
  • 5.5%–38.5% mean squared error reductions are reported across transformer architectures for the proposed multi-scale framework.The introduction also describes ablation studies supporting its architectural and methodological choices.Source: §1 · p. 2

2 RELATED WORKS

Related work spans forecasting models, transformer architectures, and multi-scale approaches. The paper positions its framework as model-agnostic and applicable to transformer-based time-series forecasting.

2 RELATED WORKS

  • Earlier forecasting approaches included ARIMA, exponential smoothing, RNNs, and temporal convolutional networks [Bai et al., 2018].The passage also notes a broad range of time-series forecasting models.Source: §2 · p. 2
  • Time-series transformers use self-attention; Informer reduces its time and memory complexity to O(L log L), while Autoformer and FEDformer introduce alternative sequence-processing mechanisms.Informer uses ProbSparse attention; Autoformer uses cross-correlation attention, and FEDformer uses frequency transforms.Source: §2 · p. 2
  • Multi-scale methods span vision, language, and time-series tasks; some time-series approaches use parallel transformations, while many related architectures propose RNN-based modules.The passage also describes multi-scale segment-wise correlations for attention.Source: §2 · p. 2
  • The framework applies successive transformer modules to refine forecasts at different temporal scales, processing from the smallest scale to the original scale.It is presented as a model-agnostic framework that keeps parameter count and time complexity roughly the same.Source: Figure 2 · p. 3, §2 · p. 3

3 METHOD

Scaleformer iteratively refines forecasts across temporal scales using a shared transformer module, with cross-scale normalization to address distribution shifts between steps. It also adapts input embeddings and uses an adaptive loss for forecasting.

3.1 PROBLEM SETTING

  • Forecasting predicts a horizon window from a preceding look-back window.Source: §3.1 · p. 3

3.2 MULTI-SCALE FRAMEWORK

  • At successive scales, the same transformer module refines the forecast using downsampled look-back inputs and an upsampled previous forecast.The look-back input is average-pooled, and the previous forecast is linearly interpolated before the next step.Source: §3.2 · p. 3

3.3 CROSS-SCALE NORMALIZATION

  • Cross-scale normalization uses a moving average of look-back and forecast statistics to mitigate distribution shifts between consecutive scales.Normalizing only by look-back or previous-forecast statistics can accumulate errors across steps.Source: §3.3 · p. 5

3.4 INPUT EMBEDDING

  • Inputs combine value, temporal, and fixed positional embeddings, with indicators for observation source and current scale.The positional embedding is adapted to the different scales.Source: §3.4 · p. 5

3.5 LOSS FUNCTION

  • The multi-scale framework with adaptive loss outperforms vanilla baselines in almost all datasets and settings.Table 1 compares MSE and MAE in multivariate settings across forecast horizons.Source: Table 1 · p. 6
  • The adaptive loss addresses outlier sensitivity, with its parameters learned end-to-end during training.The authors report this as the first adaptation of the objective to time-series forecasting.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

The experiments compare Scaleformer with transformer baselines, examine its components and qualitative forecasts, and test extensions to probabilistic forecasting and non-transformer encoders.

4.1 MAIN RESULTS

  • The evaluation uses transformer baselines and public forecasting datasets, with input length 32 and horizons 24, 32, 48, and 64.The baselines include FEDformer, Reformer, Performer, Informer, and Autoformer; the datasets include ECL, Traffic, Weather, Exchange-Rate, and National Illness.Source: §4.1 · pp. 6–7
  • 5.6% over FEDFormer, 13% over Autoformer, and 38% over Informer are the average MSE improvements reported for the proposed framework.On exchange-rate forecasting, the approach improves over Informer and Reformer by over 50% averaged across horizon lengths; the framework uses the same number of parameters except for two adaptive-loss parameters, with a small computation-efficiency trade-off.Source: §4.1 · p. 7

4.2 ABLATION STUDY

  • Combining the multi-scale framework with adaptive loss gives the best performance, while each component improves performance both with and without the other.The ablations compare adaptive loss with MSE and include variants with and without the multi-scale framework.Source: Figure 4 · p. 7, §4.2 · p. 8
  • Without cross-scale normalization, the multi-scale framework mostly has higher errors than baselines; normalization improves Informer and FEDformer, but not Autoformer.The authors attribute the Autoformer result to its internal trend-cycle normalization component.Source: Table 2 · p. 7, §4.2 · p. 8, Table 3 · p. 8

4.3 QUALITATIVE RESULTS

  • Qualitative examples show Scaleformer-based forecasts capturing trends and local signal variations better than their baselines.The examples are randomly selected, and the displayed comparison includes Informer and FEDformer.Source: §4.3 · p. 8, Figure 5 · p. 9

4.4 EXTENSIONS AND DISCUSSION

  • Scaleformer improves probabilistic forecasting with DeepAR-style outputs, using mean and standard-deviation heads trained with negative log likelihood.The authors report that Scaleformer outperforms probabilistic Informer, with NLL and CRPS as evaluation metrics.Source: §4.4 · p. 9
  • Adapting NHits and FiLM to iteratively refine predictions yields a statistically significant average improvement.These encoders use fixed-length univariate inputs and outputs; the authors describe extensions to probabilistic and non-transformer settings as promising future work.Source: §4.4 · pp. 8–9

5 CONCLUSION

The conclusion presents Scaleformer as a multi-scale prior for transformer forecasting, with iterative refinement and normalization as its core design.

5 CONCLUSION

  • The framework iteratively refines forecasts at increasingly fine-grained scales and introduces normalization to minimize distribution shifts between scales.The conclusion reports improved performance over baseline transformer architectures and forecasts that better capture trends and local variations.Source: §5 · p. 9
  • Extending Scaleformer to probabilistic forecasting and non-transformer models is identified as promising future work.Source: §5 · p. 10

A IMPLEMENTATION DETAILS

The appendix describes implementation choices and motivates multi-scale modeling through dependencies across time scales and prior multi-scale approaches.

A IMPLEMENTATION DETAILS

  • The implementation fixes the temporal scale factor at 2, uses a look-back window of 96 and horizons from 96 to 720, and repeats experiments five times.The models use hidden dimension 512, batch size 32, and Adam; Autoformer’s decoder input is adapted to make prior predictions available.Source: A IMPLEMENTATION DETAILS · p. 13

B MORE MOTIVATIONS

  • The Fraser River example illustrates inter-related annual averages, while the paper distinguishes its transformer multi-scale prior from earlier approaches using simpler statistical base models.The authors state that their work is the first to explicitly consider a multi-scale prior by construction for transformers.Source: B MORE MOTIVATIONS · p. 13

C REDUCING COMPUTATIONAL COST

The interpolated multi-scale variant reduces compute while outperforming vanilla baselines, though it trails full Scaleformer. Runtime is roughly twice the baseline at scale factor 2, and the adaptive loss is motivated by its treatment of outliers.

C REDUCING COMPUTATIONAL COST

  • With scale factor 2, Scaleformer takes roughly twice the baseline runtime, consistent with the geometric-progression estimate.The reported measurements use batch size 32 on a GeForce GTX 1080 Ti; the code was not optimized.Source: C REDUCING COMPUTATIONAL COST · p. 13, Table 4 · p. 14
  • Interpolation reduces compute and beats vanilla Informer and Autoformer in almost all settings, but performs worse than full Scaleformer.This offers a trade-off between computational efficiency and the full method’s further performance improvement.Source: C REDUCING COMPUTATIONAL COST · p. 14, Table 5 · p. 14

D JUSTIFICATION FOR THE ADAPTIVE LOSS

  • For large prediction errors ξ, the adaptive loss behaves asymptotically like ξ^α, so larger α penalizes outliers more.The authors consequently expect lower learned α where there are fewer outliers.Source: D JUSTIFICATION FOR THE ADAPTIVE LOSS · p. 14

E INTERPLAY BETWEEN ADAPTIVE LOSS AND MULTI-SCALE ARCHITECTURE

The multi-scale framework outperforms same-scale iterative refinement, and its combination with adaptive loss is presented as addressing error accumulation during sequential computation. The discussion also reports normalization benefits beyond multi-scale models.

E INTERPLAY BETWEEN ADAPTIVE LOSS AND MULTI-SCALE ARCHITECTURE

  • Sequential multi-scale computation may accumulate errors, and the authors describe adaptive loss as mitigating this issue for more stable learning.They frame the multi-scale architecture as a useful structural prior that also increases the likelihood of explosive error accumulation.Source: E INTERPLAY BETWEEN ADAPTIVE LOSS AND… · pp. 14–15

F SINGLE-SCALE MODEL WITH MEAN-NORMALIZATION

  • Mean normalization alone makes a single-scale model perform better than a multi-scale model without normalization.The authors attribute normalization’s broader benefit to distribution shifts beyond those induced by multi-scale processing, including lookback-to-forecast shifts.Source: F SINGLE-SCALE MODEL WITH… · p. 15

G.1 ITERATIVE REFINEMENT USING THE SAME SCALE

  • The multi-scale framework outperforms same-scale iterative refinement while incurring significantly lower memory and time complexity overhead.The comparison uses scale factor 2 for Scaleformer and scale factor 1 for five same-scale refinement iterations.Source: §G.1 · p. 15, Table 6 · p. 15

G.2 MORE RESULTS ON DIFFERENT COMPONENTS

  • Combining adaptive loss with the multi-scale framework significantly improves performance for both Informer and Autoformer.The component comparison includes adaptive loss alone, multi-scale with MSE, and their combination.Source: Table 7 · p. 16

G.3 RESULTS ON DIFFERENT SCALES

Scale factor 2 performs best among the reported scale choices. Additional analyses relate learned α to outliers, show qualitative correction across scales, and discuss why normalization alone offers limited gains for Autoformer.

G.3 RESULTS ON DIFFERENT SCALES

  • Scale factor 2 achieves better performance than the other reported scale choices.The comparison describes reducing the scale factor from 16 to 2 while increasing the number of steps.Source: §G.3 · p. 17, Table 8 · p. 17

G.4 EFFECT OF α ON THE LOSS FUNCTION

  • Weather has the lowest learned α and the largest relative outlier scales among the illustrated datasets.The analysis associates lower α with greater robustness to outliers and excludes c because it does not affect that robustness.Source: §G.4 · p. 18, Figure 6 · p. 18
  • Qualitative results show that the model can correct mistakes from earlier scales, including forecasting artifacts occurring at individual scales.Source: Figure 7 · p. 18

H DISCUSSION ON NORMALIZATION FOR AUTOFORMER

  • Autoformer gains little from normalization alone, consistent with its series-decomposition module already reducing internal distribution shift.The authors contrast this with the proposed normalization’s benefits to other baselines.Source: H DISCUSSION ON NORMALIZATION FOR… · p. 19

I ADDITIONAL QUALITATIVE RESULTS

Additional qualitative comparisons report that Scaleformer better learns local and global time-series variations. A probabilistic Informer adaptation also shows benefits on probabilistic metrics, using prediction heads for μ and σ.

I ADDITIONAL QUALITATIVE RESULTS

J PROBABILISTIC FORECASTING

  • The probabilistic Informer adaptation uses separate prediction heads for μ and σ, with negative log likelihood training and NLL and CRPS evaluation.Source: J PROBABILISTIC FORECASTING · p. 20
  • Scaleformer improves probabilistic metrics for Informer in the reported comparison, though dedicated probabilistic approaches are excluded.Source: Table 9 · p. 20

K EXTENSION TO OTHER METHODS

The framework is extended to non-transformer models, while cross-scale normalization balances internal distribution shifts against information loss. Normalization results vary with dataset distributions.

K EXTENSION TO OTHER METHODS

  • The framework is applied to NHiTS and FiLM by using separate model copies across scales and concatenating each scale’s input with the previous scale’s output.The comparison uses the same optimizer and learning rate as the prior baselines.Source: K EXTENSION TO OTHER METHODS · p. 21
  • Table 10 compares the framework’s effect on NHiTS and FiLM, two non-transformer models.The caption says best results are shown in bold but provides no values here.Source: Table 10 · p. 21

L CROSS-SCALE NORMALIZATION

  • Cross-scale normalization balances distribution-shift reduction against information loss from removing input mean and standard deviation information.The authors attribute mean-only normalization’s advantage to retaining more information while still addressing internal distribution shifts.Source: L CROSS-SCALE NORMALIZATION · p. 21
  • Mean normalization works better on nearly unimodal Electricity and Traffic data, while mean-and-standard-deviation normalization works better on multimodal Exchange Rate and, to some extent, Weather data.Both normalization forms improve substantially over no normalization; Figure 10 shows dataset distributions.Source: L CROSS-SCALE NORMALIZATION · p. 21, Figure 10 · p. 21

M SYNTHETIC DATASET

On a synthetic dataset built from Mackey–Glass series, the proposed framework significantly improves performance over baselines. Adaptive-loss gains grow with outlier prevalence, and statistical tests assess gains from both adaptive loss and multi-scale architectures.

M SYNTHETIC DATASET

  • Zero-mean shifting performs better on almost all Traffic, Electricity, and Illness results, while standard normalization performs better on Weather and Exchange.The authors use zero-mean shifting because it removes less information from the input time series.Source: Table 11 · p. 22
  • The proposed framework significantly improves over baselines on a synthetic dataset combining three Mackey–Glass series generated with τ = 18, 12, and 9.Table 12 compares the framework and baselines on this dataset.Source: M SYNTHETIC DATASET · p. 22, Table 12 · p. 22
  • With extreme outliers in 5% of training data, adaptive loss improves performance by roughly 70% at most, while matching MSE on clean data.The paper defines an outlier as a sample more than 50 times the series’ standard deviation from its median; Figure 11 shows improvement increasing with outlier percentage.Source: Figure 11 · p. 22, M SYNTHETIC DATASET · p. 23

N STATISTICAL TESTS

  • Student’s t-tests report adaptive-loss gains as significant for most settings (p < 0.05), and multi-scale gains as almost always significant, with some weather and exchange-rate window exceptions.The adaptive-loss exceptions include Exchange Rate and certain Traffic window sizes; the paper reports p < 0.5 for multi-scale gains in most settings.Source: N STATISTICAL TESTS · p. 23, Table 13 · p. 23, Table 14 · p. 23
Loading 2206.04038v4…