PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Multi-Scale Decomposition Framework for Time Series Forecasting

Yifan Hu, Peiyuan Liu, Peng Zhu, Dawei Cheng, Tao Dai

arXiv:2406.03751v2cs.LG

TL;DR

Forecasting models must capture temporal patterns that differ across scales and contribute unequally, while existing Transformer and MLP approaches have distinct efficiency and representation challenges. AMD decomposes series at multiple scales, models temporal and channel dependencies, and adaptively integrates patterns; the paper reports state-of-the-art results for long- and short-term forecasting across datasets with superior efficiency.

  • Problem

    Forecasting must model temporal patterns that vary across scales and contribute nonuniformly, while Transformer methods can be computationally costly and MLPs can struggle with diverse patterns.

    Source: Introduction · p. 1

  • Method

    AMD decomposes series into multiple temporal patterns, models temporal and channel dependencies, and adaptively integrates the patterns using MDM, DDI, and AMS blocks.

    Source: Introduction · p. 2

  • Results

    AMD reports state-of-the-art performance in long- and short-term forecasting across datasets, with superior efficiency.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The reported scope spans both long- and short-term forecasting across multiple datasets.

    Source: Conclusion · p. 7

    AMD's patching mechanism extracts recent history, but better exploitation of locality remains an issue for further research.

    Source: Improvement · p. 14

Abstract

from arXiv · show

Transformer-based and MLP-based methods have emerged as leading approaches in time series forecasting (TSF). While Transformer-based methods excel in capturing long-range dependencies, they suffer from high computational complexities and tend to overfit. Conversely, MLP-based methods offer computational efficiency and adeptness in modeling temporal dynamics, but they struggle with capturing complex temporal patterns effectively. To address these challenges, we propose a novel MLP-based Adaptive Multi-Scale Decomposition (AMD) framework for TSF. Our framework decomposes time series into distinct temporal patterns at multiple scales, leveraging the Multi-Scale Decomposable Mixing (MDM) block to dissect and aggregate these patterns in a residual manner. Complemented by the Dual Dependency Interaction (DDI) block and the Adaptive Multi-predictor Synthesis (AMS) block, our approach effectively models both temporal and channel dependencies and utilizes autocorrelation to refine multi-scale data integration. Comprehensive experiments demonstrate that our AMD framework not only overcomes the limitations of existing methods but also consistently achieves state-of-the-art performance in both long-term and short-term forecasting tasks across various datasets, showcasing superior efficiency. Code is available at https://github.com/TROUBADOUR000/AMD

Introduction

Time series forecasting must handle temporal patterns that vary across scales, but Transformers can be costly and overfit while MLPs can miss complex patterns. AMD decomposes series across scales, models temporal and channel dependencies, and adaptively integrates the patterns.

Introduction

  • Temporal patterns differ across sampling scales, and their contributions to future variation are nonuniform and jointly determined.The text contrasts fine-grained hourly weather changes with coarse monthly climate variation.Source: Introduction · p. 1
  • Transformers can incur quadratic sequence-length complexity and overfit, while simple MLP mappings can bottleneck representation of diverse temporal patterns.Transformers may overemphasize mutation points; MLPs are computationally efficient and model temporal dynamics chronologically.Source: Introduction · pp. 1–2
  • AMD uses MDM for residual multi-scale decomposition and aggregation, DDI for temporal and channel dependencies, and AMS for adaptive integration of the resulting patterns.The framework also uses autocorrelation to model scale contributions and differs from average aggregation.Source: Introduction · p. 2
  • The authors report theoretical analysis demonstrating the feasibility of decomposing and adaptively aggregating temporal patterns across scales.The contribution statement describes identifying dominant intertwined patterns and accounting for their varied impacts on future forecasts.Source: Introduction · p. 2
  • AMD reports state-of-the-art performance in long- and short-term forecasting across datasets, with superior efficiency.The introduction describes the framework as consistently achieving these results.Source: Introduction · p. 2

Time Series Forecasting

  • The paper positions AMD's multi-scale approach as a response to MLP representational limits and to Transformer efficiency problems in time-series forecasting.The related-work discussion contrasts these model families and describes AMD's separate linear models for different scales.Source: Introduction · p. 1, Time Series Forecasting · p. 2

Mixture of Experts for TSF

  • AMD applies mixture-of-experts adaptability by customizing predictors for individual temporal patterns.The paper contrasts this approach with earlier TSF uses of experts for horizons or trend components.Source: Mixture of Experts for TSF · p. 2

Series Decomposition in TSF

  • The paper identifies ignored high-order interactions across scales as a limitation of existing decomposition designs and proposes adaptive decomposition to model dynamic interactions.The related-work passage describes seasonal-trend, frequency-based, hierarchical, and multi-scale approaches.Source: Series Decomposition in TSF · pp. 2–3

Preliminary: Linear Models with Multi-Scale Information for TSF

The preliminary section formulates forecasting as predicting future target variables from historical observations, then introduces multi-scale information through downsampling. Under smoothness conditions, it states that linear models can use this information with bounded error for nonperiodic patterns.

Preliminary: Linear Models with Multi-Scale Information for TSF

  • The forecasting setup maps historical observations to future values, with a linear model using learned parameters to produce predictions.The passage defines the input by its variable count and look-back length, and the target by its variable count and forecast horizon.Source: Preliminary: Linear Models with… · p. 3
  • The multi-scale construction assumes a smooth original sequence and creates coarser sequences through repeated downsampling at rate d.The text also notes that smoothness or periodicity is relevant to predictability in its motivating real-world applications.Source: Preliminary: Linear Models with… · p. 3
  • For smooth nonperiodic patterns, the theoretical analysis states that linear models can use multi-scale information with bounded prediction error.The theorem passage specifies a Lipschitz-smooth original sequence and states that the error is bounded across prediction steps.Source: Preliminary: Linear Models with… · p. 3

Method

AMD addresses forecasting’s entangled multi-scale temporal patterns through decomposition and autocorrelation, using MDM, DDI, and AMS to mix scales, model dependencies, and synthesize predictions.

Method

  • AMD decomposes time series into multiple scales, models their temporal and channel interactions, and adaptively aggregates predictions to address patterns’ varying influence on forecasts.Its three components are MDM, DDI, and AMS.Source: Method · p. 3

Multi-Scale Decomposable Mixing Block

  • MDM extracts coarse patterns by average pooling, then mixes patterns from coarse to fine through a feedforward residual network to produce channel-level information.The raw input supplies fine-grained details; the coarser scales are pooled from preceding patterns.Source: Multi-Scale Decomposable Mixing Block · p. 3

Dual Dependency Interaction Block

  • DDI applies time-mixing and channel-mixing to aggregated multi-scale information, while scaling rate β can suppress unwanted cross-channel interference.Cross-channel dependencies may introduce interference when target series are not correlated with other covariates.Source: Dual Dependency Interaction Block · pp. 3–4

Adaptive Multi-predictor Synthesis Block

  • AMS selects dominant temporal patterns and weights predictor outputs to form each channel’s prediction; its selector untangles intertwined scales, while dense MoE uses every pattern.The selector generates weights, and the projection combines predictions using those weights.Source: Adaptive Multi-predictor Synthesis Block · p. 4

Loss Function

  • AMS trains predictors with MSE and adds a selector coefficient-of-variation loss to encourage balanced expert assignments, alongside L2 regularization.The total loss combines predictor loss, weighted selector loss, and weighted parameter regularization.Source: Loss Function · p. 5

Experiments

AMD is evaluated on long-term and short-term forecasting, with additional analyses of its components, efficiency, and behavior. It performs strongly across benchmark cases, while ablations examine multi-scale inputs, dependency modeling, and predictor weighting.

Main Results

  • AMD achieves the best performance in 50 cases and the second-best in 27 of 80 overall cases, across high- and low-dimensional datasets.The benchmarks cover long-term forecasting datasets and PEMS short-term forecasting tasks.Source: Main Results · pp. 5–6

Model Analysis

  • Removing MDM increases prediction error, indicating that relying on a single scale is insufficient and omitting non-dominant patterns degrades performance.The component ablation is conducted on Weather and ECL.Source: Model Analysis · pp. 6–7
  • Cross-channel dependencies do not always improve accuracy; scaling rate β balances their emphasis against temporal dependencies and can boost performance.The authors report this pattern across other datasets as well.Source: Model Analysis · p. 6
  • Randomized pattern order, equal weighting, and removing AMS each reduce performance; AMS preserves sequential order and adaptively weights changing dominant patterns.The authors contrast this with permutation-invariant self-attention and simple averaging.Source: Model Analysis · pp. 6–7
  • AMD assigns greater weight to TP16 before T and TP5 after T, while cross-channel dependencies can smooth target variability and shift its distribution when covariates are uncorrelated.These behaviors are illustrated in Figures 3 and 4.Source: Figure 3 · p. 6, Model Analysis · p. 7, Figure 4 · p. 7
  • On Weather, AMD is reported as more efficient than Transformer- and MLP-based models with relatively few parameters; adding predictors lowers MSE but increases time and memory.The experiments use input length 512 and prediction length 96; the authors use eight predictors to balance these trade-offs.Source: Model Analysis · p. 7, Figure 5 · p. 7
  • Using the TP-Selector load-balancing loss improves MSE by more than 11.2% compared with λ1 = 0.0.Selector weights also do not tend to favor specific temporal patterns, addressing load imbalance in sparse MoE structures.Source: Model Analysis · p. 7
  • Adding MDM and AMS improves DLinear and MTS-Mixers while maintaining the same computational resource requirements.The comparison reports the average percentage improvement of MDM and AMS over the original methods.Source: Model Analysis · p. 7, Table 4 · p. 7

Conclusion

The conclusion presents AMD as a framework that decomposes time series into patterns at multiple scales and adaptively aggregates them. The authors report state-of-the-art performance in long- and short-term forecasting across datasets, with superior efficiency and effectiveness.

Conclusion

  • AMD decomposes time series into multiple-scale temporal patterns and adaptively aggregates them, achieving reported state-of-the-art long- and short-term forecasting performance across datasets.The conclusion also characterizes the framework as having superior efficiency and effectiveness.Source: Conclusion · p. 7

Proof of Theorem 1

The proof considers a Lipschitz-smooth original sequence and derives bounded prediction error for the weighted pattern predictor. It also establishes smoothness for downsampled sequences and multi-scale mixing representations.

Proof of Theorem 1

  • The theorem considers a Lipschitz-smooth original sequence with constant K and a multi-scale mixing representation.The proof simplifies to univariate sequences; the finest-granularity pattern has period P, which tends to infinity when there is no periodicity.Source: Proof of Theorem 1 · pp. 9–10
  • The proof establishes that downsampled sequences and multi-scale mixing representations preserve Lipschitz smoothness.It proceeds by induction over downsampling and mixing operations.Source: Proof of Theorem 1 · pp. 9–10
  • Internal averaging in the weighted pattern predictor keeps the resulting data bounded, so the prediction error |yt − ˆyt| is bounded.The stated conclusion is that a linear model exists with bounded error for all t = 1, ..., T.Source: Proof of Theorem 1 · pp. 9–10

Detailed Algorithm Description

The algorithm normalizes the input, extracts and aggregates multi-scale information, then predicts from feature-specific weighted outputs before returning de-normalized predictions.

Detailed Algorithm Description

  • The algorithm initializes data and parameters, normalizes the input, and iterates through MDM to extract multi-scale information.This is the opening stage of the stated AMD algorithm.Source: Detailed Algorithm Description · p. 10
  • DDI blocks perform aggregation, after which TPSelector determines predictor weights for each feature and the weighted outputs are concatenated for prediction.The procedure applies the selector separately for each feature.Source: Detailed Algorithm Description · p. 10
  • The algorithm transposes and de-normalizes the predictions before returning them.Source: Detailed Algorithm Description · p. 10

Details of Experiments

The experiments describe the datasets and baselines, specify forecasting metrics, and report the implementation settings used across datasets.

Detailed Dataset Descriptions

Baseline Models

  • The selected baselines span MLP, Transformer, and CNN approaches, including MTS-Mixers, DLinear, TiDE, PatchTST, iTransformer, Crossformer, FEDformer, TimesNet, and MICN.The cited baseline descriptions characterize MTS-Mixers as using two factorized modules and DLinear as a one-linear-layer model.Source: Baseline Models · p. 11

Metric Details

  • Long-term forecasting is evaluated using mean square error (MSE) and mean absolute error (MAE).Table 6 lists hyper-parameters for different experimental settings.Source: Metric Details · p. 11, Table 6 · p. 12

Hyper-Parameter Selection and Implementation Details

  • AMD uses eight predictors, topK 2, three MDM layers, and downsampling rate 2 across datasets; baseline implementations follow original-paper or official-code configurations.Other settings include weight decay 1e−7, Adam optimization, and AMS hidden dimension 2048.Source: Hyper-Parameter Selection and… · p. 12

Extra Experimental Results

The reported robustness evaluation finds stable AMD performance across five random seeds. Sensitivity experiments examine downsampling, predictor count, and selector-loss scaling to choose settings balancing performance, memory, and efficiency.

Robustness Evaluation

  • AMD's performance is stable across five random seeds on ETTh1, ETTm1, Weather, Solar-Energy, ECL, and Traffic.The results are reported in Table 7.Source: Robustness Evaluation · p. 12

Hyper-Parameter Sensitivity

  • Increasing downsampling operations improves results at different prediction lengths, and three layers are chosen to balance efficiency and performance.The comparison varies downsampling operations h and rate d on ETTm1; Table 8 reports MSE results.Source: Hyper-Parameter Sensitivity · p. 12
  • On Weather, increasing predictor count improves prediction results but raises selector-weight memory use, motivating the choice of eight predictors.Figure 6 includes sensitivity results for predictor count.Source: Hyper-Parameter Sensitivity · p. 12
  • On Weather, λ1=0 yields poor prediction because the selector loss does not guide reasonable pattern assignments; nonzero settings are robust, so λ1=1.0 is selected.Figure 6 includes sensitivity results for λ1.Source: Hyper-Parameter Sensitivity · p. 12

Improvement

The section discusses sequence-modeling designs in AMD, including normalization, patching, frequency representations, channel dependencies, and adaptive decomposition. It also identifies open issues in locality, distributional shift, generalization, and data quality.

Normalization

  • AMD adopts RevIn normalization and denormalization, but this does not adequately resolve intricate distribution shifts among deep-network layers.RevIn constrains input mean and variance and restores them at the output.Source: Improvement · p. 12

Patching

  • AMD uses patching to extract recent-history information, while better exploitation of time-series locality remains an open issue.Figure 7 includes patching among the sequential-modeling designs.Source: Figure 7 · p. 13, Improvement · p. 14

Frequency Domain Representation

  • Frequency-domain methods can rely on engineered dominant periods, potentially overlooking periods or trend changes and losing information.The authors propose exploring adaptive temporal-pattern mining in the frequency domain.Source: Improvement · p. 14

Channel Independence and Channel Dependence

  • AMD combines channel independence with dependencies across variables over time, while the capacity–robustness trade-off and its relation to generalization remain open questions.Figure 7 includes channel independence among the sequential-modeling designs.Source: Figure 7 · p. 13, Improvement · p. 14

Sequence Decomposition

  • AMD's adaptive decomposition goes beyond seasonal-and-trend splits, but its effectiveness depends heavily on historical-data availability and quality.Figure 7 frames sequence decomposition among five sequential-modeling designs.Source: Figure 7 · p. 13, Improvement · p. 14

Future Directions

  • The paper identifies distribution shift and multivariate sequence modeling as issues for future sequential-modeling designs.The authors expect further designs to address these issues.Source: Improvement · p. 14

Related papers

Loading 2406.03751v2…