PaperlayerView exact revision on arXiv

Source-linked AI summary

Amplifier: Bringing Attention to Neglected Low-Energy Components in Time Series Forecasting

Jingru Fei, Kun Yi, Wei Fan, Qi Zhang, Zhendong Niu

arXiv:2501.17216v3cs.LG

TL;DR

Existing forecasting models can neglect low-energy components, which the paper identifies as an issue for time-series forecasting. Amplifier boosts these components and restores their energy, combining the technique with a seasonal-trend forecaster and an optional channel-interaction block; the authors report strong results across eight datasets.

  • Problem

    Existing forecasting models tend to neglect low-energy components, which the paper identifies as an issue for time-series forecasting.

    Source: §1 · p. 1

  • Method

    Amplifier boosts low-energy components and restores their energy, models two energy peaks separately with a seasonal-trend forecaster, and includes an SCI block for channel relationships.

    Source: §1 · p. 2

  • Results

    Across eight real-world datasets, Amplifier consistently outperforms state-of-the-art methods while offering higher efficiency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The energy amplification technique is presented as a general approach for improving other time-series forecasting models.

    Source: §1 · p. 2

    Compared with RLinear and DLinear, Amplifier has a larger parameter size and longer training time.

    Source: Efficiency Analysis · p. 7

Abstract

from arXiv · show

We propose an energy amplification technique to address the issue that existing models easily overlook low-energy components in time series forecasting. This technique comprises an energy amplification block and an energy restoration block. The energy amplification block enhances the energy of low-energy components to improve the model's learning efficiency for these components, while the energy restoration block returns the energy to its original level. Moreover, considering that the energy-amplified data typically displays two distinct energy peaks in the frequency spectrum, we integrate the energy amplification technique with a seasonal-trend forecaster to model the temporal relationships of these two peaks independently, serving as the backbone for our proposed model, Amplifier. Additionally, we propose a semi-channel interaction temporal relationship enhancement block for Amplifier, which enhances the model's ability to capture temporal relationships from the perspective of the commonality and specificity of each channel in the data. Extensive experiments on eight time series forecasting benchmarks consistently demonstrate our model's superiority in both effectiveness and efficiency compared to state-of-the-art methods.

1 Introduction

The introduction identifies neglect of low-energy frequency components as a forecasting-model issue, with filtering them associated with increased error. Amplifier addresses it by boosting these components before learning and restoring their energy, within a model that separately models two energy peaks.

1 Introduction

  • Existing deep-learning forecasters tend to focus on high-energy components and neglect low-energy components, limiting use of informative aspects of the series.The reported pattern of ignored components follows energy magnitude regardless of their frequency-band position.Source: §1 · pp. 1–2
  • Filtering low-energy components increases MSE across PatchTST, RLinear, DLinear, and FreTS.The introduction presents this as empirical evidence that these components matter for prediction accuracy.Source: §1 · p. 2
  • The Energy Amplification Technique boosts low-energy components to improve learning efficiency, then restores their energy to its original level.The paper describes the technique as applicable to other time-series forecasting models.Source: §1 · p. 2
  • Amplifier combines energy amplification with a seasonal-trend forecaster to model the two energy peaks separately, and adds an optional SCI block for channel commonality and specificity.The amplified data exhibits two spectral energy peaks; the seasonal and trend components correspond to them.Source: §1 · p. 2
  • Across eight real-world datasets, Amplifier consistently outperforms state-of-the-art methods while offering higher efficiency.The paper presents energy amplification as a general technique that can also be applied to other forecasting models.Source: §1 · p. 2

2 Related Work

Related work describes a broad landscape of deep-learning forecasters, alongside frequency-domain methods for improving forecasting. The paper positions Amplifier as a unified approach for Transformer and MLP networks that targets neglected low-energy components.

Deep Time Series Forecasting

  • Deep time-series forecasting includes RNN-, TCN-, Transformer-, and linear-based methods.Examples include LSTNet and DeepAR, SCINet and TimesNet, PatchTST and iTransformer, and DLinear and RLinear.Source: Deep Time Series Forecasting · p. 2

Advancements in Frequency Domain Techniques

  • Frequency-domain methods use techniques such as low-rank attention, interpolation, and frequency-domain MLPs, but may treat energy levels uniformly or discard high-frequency components.The paper describes the latter as an oversight that often leads to sub-optimal forecasting performance.Source: Advancements in Frequency Domain… · p. 2
  • Fredformer addresses frequency bias by learning across frequency bands, while Amplifier is presented as a unified method for Transformer and MLP networks.The paper also reports that Amplifier trains faster and has a smaller parameter scale than Fredformer.Source: Advancements in Frequency Domain… · pp. 2–3

3 Preliminaries

The preliminaries frame low-energy components as underlearned parts of the frequency spectrum and analyze how energy relates to training losses and parameter updates. They define forecasting from historical inputs and express its loss as separate high- and low-energy contributions.

3 Preliminaries

  • The paper distinguishes low- and high-energy frequency components in the spectrum of each time series.It notes that real-world spectra often show a clear distinction between high- and low-energy levels.Source: §3 · p. 3
  • Forecasting predicts the next τ timestamps from historical multivariate input, with the loss expressible as the sum of high- and low-energy component losses.The frequency-domain expression separates the losses and their associated parameters by energy level.Source: §3 · p. 3
  • At the initial training stage, high-energy components contribute substantially more to total loss than low-energy components.The paper argues this can direct training toward reducing high-energy error and potentially neglect low-energy components.Source: §3 · p. 3
  • Parameters associated with low-energy components update more slowly and less efficiently than those associated with high-energy components.By the end of training, the low-energy-associated parameters remain farther from their true values, which the paper links to their minimal participation.Source: §3 · p. 3

4 Methodology

Amplifier boosts low-energy components, restores their energy after forecasting, and combines this technique with channel-pattern modeling and seasonal-trend prediction.

4 Methodology

  • The proposed general energy amplification technique boosts low-energy components and restores their energy after modeling.The authors also introduce Amplifier as a forecasting model built around this technique.Source: §4 · p. 3

Energy Amplification Technique

  • Spectrum flipping transfers energy from the low-frequency region to high-frequency low-energy components, making their energy comparable to high-energy components.The block adds the flipped spectrum to the original and applies an inverse discrete Fourier transform.Source: Energy Amplification Technique · pp. 3–4
  • The restoration block estimates the additional spectrum from the flipped spectrum, removes it from the amplified prediction, and returns the result to the time domain.It first adjusts the input length to match the prediction length.Source: Energy Amplification Technique · p. 4

Amplifier

  • SCI separates shared channel patterns from channel-specific patterns, predicts the common pattern, and combines it with the specific patterns.The commonality is represented as an abstract main channel.Source: Amplifier · pp. 4–5
  • The seasonal-trend forecaster decomposes the SCI output into seasonal and trend components and predicts them with separate feed-forward networks.The resulting seasonal and trend predictions are combined as the block output.Source: Amplifier · p. 5

5 Experiments

Across eight datasets, Amplifier leads on most forecasting comparisons; experiments also report gains from energy amplification and SCI, alongside efficiency trade-offs and visual prediction comparisons.

Experiments Setup

  • Experiments covered eight datasets, ten forecasting baselines, and reported MSE and MAE, with MSE used as the loss function.Experiments ran in PyTorch on one NVIDIA RTX 3070 GPU with 8GB.Source: Experiments Setup · p. 5

Main Results

  • At lookback 96, Amplifier earned 11 top-1 and 2 top-2 positions out of 16; at lookback 336, it retained exceptional predictive performance.The lookback-336 comparison covers prediction lengths 96, 192, 336, and 720.Source: Main Results · p. 5, Table 3 · p. 6

Model Analysis

  • Figure 3 presents ablation prediction results for the energy amplification technique in four representative models, identifying the evaluated method and model count without specifying numerical outcomes in the caption.Source: Figure 3 · p. 6
  • On Weather, adding energy amplification affected MSE by 15.100% and MAE by 10.923% compared with Amplifier without EAT.The ablation compares Amplifier with its version without the energy amplification technique.Source: Table 2 · p. 5, Model Analysis · p. 6
  • Adding the SCI block improved MSE and MAE on both Electricity and Traffic, where channel interactions were tested.On Electricity, improvements were 7.479% in MSE and 3.007% in MAE; on Traffic, 5.018% in MSE and 4.294% in MAE.Source: Table 4 · p. 6, Model Analysis · p. 6

Efficiency Analysis

  • Amplifier has medium-scale parameters and training speed, but is larger and slower than RLinear and DLinear.Its theoretical complexity is O(L log L); the comparison uses Weather with L=96 and τ=96.Source: Efficiency Analysis · pp. 6–7, Figure 4 · p. 7

Visualizations

  • On two synthetic signals, Amplifier modeled low-energy components as effectively as high-energy ones, unlike four compared models across low- and high-frequency regions.The comparison includes iTransformer, PatchTST, RLinear, and DLinear.Source: Figure 5 · p. 7, Visualizations · p. 7
  • On ETTm2 and Electricity visualizations, Amplifier provided the most accurate future-series predictions among the compared models.The plots distinguish input sequence, ground truth, and prediction with red, blue, and yellow lines, respectively.Source: Figure 6 · p. 7, Figure 7 · p. 7, Visualizations · p. 7

6 Conclusion

The energy amplification technique uses amplification and restoration blocks to address the neglect of low-energy components. Amplifier applies spectrum flipping to raise those components’ energy, and experiments on eight real-world datasets validate its superiority.

6 Conclusion

  • The energy amplification technique comprises blocks that boost low-energy components and restore their energy to its original level.The boost is intended to enhance the model’s learning efficiency for those components.Source: §6 · p. 7
  • Amplifier uses spectrum flipping to raise low-energy energy levels and increase the model’s attention to those components.The paper presents Amplifier as a forecasting model built to better leverage the technique.Source: §6 · p. 7
  • Experiments on eight real-world datasets validate the superiority of Amplifier.Source: §6 · p. 7

A Theoretical Proofs

The proof represents signals as frequency components and separates their losses and parameters by energy level. It argues that high-energy components dominate early loss and receive more effective parameter updates than low-energy components.

A Theoretical Proofs

  • A signal is represented as frequency components with amplitudes and phases, and the total loss is decomposed into high- and low-energy component losses.Source: A Theoretical Proofs · p. 10
  • During initial training, the high-energy component loss occupies a significantly larger share of the overall loss than the low-energy component loss.The proof frames this comparison under the assumption that initial network parameters are typically small.Source: A Theoretical Proofs · p. 10
  • The proof derives frequency-domain parameter updates from common time-domain update algorithms and distinguishes parameters associated with high- and low-energy components.Source: A Theoretical Proofs · p. 11
  • Because low-energy amplitudes are smaller, their frequency and phase parameters receive smaller updates per iteration, reducing update efficiency across their parameter set.The theorem states that updates for low-energy parameters are much less efficient than those for high-energy components.Source: A Theoretical Proofs · p. 11

B More Explanation of Energy Amplification Technique

The energy amplification technique uses spectrum flipping to raise low-energy components to high-energy levels. The paper highlights its straightforward implementation and adaptive frequency-domain interpolation.

B More Explanation of Energy Amplification Technique

C Supplementary Details of the Experiments

The supplementary experiments describe the benchmark datasets, comparison models, and evaluation setup for Amplifier. Full multivariate results are presented for eight datasets across four prediction lengths, with averages over those lengths.

Experiment Settings

  • Experiments use the iTransformer data-processing and split protocol, PyTorch 2.0.1, one NVIDIA RTX 3070 GPU, ADAM, and L2 loss.Training is fixed to 10 epochs, with batch size in {128, 256} and FFN hidden size in {128, 256, 512}.Source: C Supplementary Details of the… · p. 11

Datasets

Baselines

Full Results

  • Table 6 presents full multivariate forecasting results for eight datasets at prediction lengths 96, 192, 336, and 720, with averages across those lengths.The table marks best results in red and second-best results in blue; the supplied passages do not report the cell values.Source: C Supplementary Details of the… · p. 12, Table 6 · p. 13

Related papers

Loading 2501.17216v3…