Multilevel Wavelet Decomposition Network for Interpretable Time Series Analysis

Jingyuan Wang; Ze Wang; Jianfeng Li; Junjie Wu

Source-linked AI summary

Multilevel Wavelet Decomposition Network for Interpretable Time Series Analysis

Jingyuan Wang, Ze Wang, Jianfeng Li, Junjie Wu

arXiv:1806.08946v1Jun 23, 2018cs.LGeess.SPstat.ML

TL;DR

The paper tackles the limited modeling of frequency information in deep time-series analysis. It introduces trainable multilevel wavelet networks and task-specific models, which outperform state-of-the-art baselines across classification and forecasting evaluations while supporting importance-based interpretation.

Problem
Wavelet preprocessing can improve neural time-series models but is loosely coupled and independently optimized, leaving integration with deep learning unresolved.
Method
mWDN learns trainable multilevel wavelet decompositions and supports RCF for classification and mLSTM for forecasting within deep learning frameworks.
Results
RCF and mLSTM show superiority to state-of-the-art baselines on 40 UCR datasets and a real-world user-volume dataset.
Takeaways & Limitations
Importance analysis identifies influential time-series elements and mWDN layers, indicating an interpretability advantage for frequency-aware deep learning.

Abstract

from arXiv · show

Recent years have witnessed the unprecedented rising of time series from almost all kindes of academic and industrial fields. Various types of deep neural network models have been introduced to time series analysis, but the important frequency information is yet lack of effective modeling. In light of this, in this paper we propose a wavelet-based neural network structure called multilevel Wavelet Decomposition Network (mWDN) for building frequency-aware deep learning models for time series analysis. mWDN preserves the advantage of multilevel discrete wavelet decomposition in frequency learning while enables the fine-tuning of all parameters under a deep neural network framework. Based on mWDN, we further propose two deep learning models called Residual Classification Flow (RCF) and multi-frequecy Long Short-Term Memory (mLSTM) for time series classification and forecasting, respectively. The two models take all or partial mWDN decomposed sub-series in different frequencies as input, and resort to the back propagation algorithm to learn all the parameters globally, which enables seamless embedding of wavelet-based frequency analysis into deep learning frameworks. Extensive experiments on 40 UCR datasets and a real-world user volume dataset demonstrate the excellent performance of our time series models based on mWDN. In particular, we propose an importance analysis method to mWDN based models, which successfully identifies those time-series elements and mWDN layers that are crucially important to time series analysis. This indeed indicates the interpretability advantage of mWDN, and can be viewed as an indepth exploration to interpretable deep learning.

1 INTRODUCTION

The paper addresses the limited integration of frequency information into deep time-series models by introducing trainable wavelet-based architectures for classification and forecasting. Experiments evaluate these models on 40 UCR datasets and a real-world user-volume dataset, alongside an importance analysis for interpretability.

Motivation: Wavelet transforms capture time- and frequency-domain features, but preprocessing-based integration with deep models is not globally optimized.The paper frames seamless integration of wavelet transforms into deep learning as a major challenge.
mWDN: mWDN hierarchically decomposes time series into frequency-ranked sub-series while allowing all parameters to be fine-tuned for different learning tasks.It combines multilevel wavelet decomposition with the parameter-learning ability of deep neural networks.
Applications: RCF uses mWDN outputs from different levels with a residual classifier stack for time series classification, while mLSTM uses decomposed sub-series for forecasting.The models target representative feature extraction in classification and hidden trends across frequencies in forecasting.
Evaluation: Experiments on 40 UCR datasets and a real-world user-volume dataset show superiority to state-of-the-art baselines and advantages from trainable mWDN parameters.The evaluation covers both time series classification and forecasting.
Interpretability: An importance analysis identifies influential time-series elements and mWDN layers, supporting the model’s interpretability advantage.The method is presented as an exploration of interpretable deep learning.

2 MODEL

The model section develops mWDN as a trainable multilevel time-frequency decomposition network, then builds RCF for classification and mLSTM for forecasting from its frequency-specific sub-series.

Multilevel Discrete Wavelet Decomposition: MDWD decomposes time series into low- and high-frequency sub-series level by level, preserving reconstruction and ordered frequency information.Higher levels increase frequency resolution while reducing time resolution, especially for low-frequency sub-series.
Multilevel Wavelet Decomposition Network: mWDN approximates MDWD within a deep neural network while allowing its initialized wavelet-filter weights to remain trainable.The network uses trainable biases and average pooling for down-sampling, with weights initialized from wavelet filter coefficients.
Residual Classification Flow: RCF feeds sub-series from multiple mWDN levels into independent classifiers and combines their outputs through residual learning.Using all levels exposes patterns at different time and frequency resolutions, while later classifiers learn incremental information.
Multi-frequency Long Short-Term Memory: mLSTM forecasts the sub-series from the final mWDN level with independent LSTM subnetworks and fuses their predictions through a fully connected network.This design separates forecasting across frequency components before ensemble fusion.

3 OPTIMIZATION

The optimization section trains RCF and mLSTM with back propagation, using task-specific objectives and regularization to preserve the wavelet decomposition prior in mWDN.

RCF optimization: RCF uses deep supervision with cross-entropy objectives for its classifiers, and the final classifier output provides the classification result.For multiple classifiers, the overall objective is a weighted sum of their objectives.
mLSTM optimization: mLSTM uses pre-training on wavelet components followed by fine-tuning against the real future value predicted by the model.Pre-training decomposes the future target into wavelet components before combining the LSTM outputs.
Joint parameter learning: Back propagation iteratively updates the parameters of both RCF and mLSTM, including the mWDN weights.Regularization terms constrain the trainable weights toward their initialized wavelet-filter structure.
Joint parameter learning: The mWDN weights converge near the wavelet-decomposition prior unless that decomposition is far inappropriate for the task.This is the stated effect of the introduced regularization terms.

4 EXPERIMENTS

The experiments evaluate mWDN-based models on time series classification and forecasting, finding strong performance against baselines and benefits from trainable wavelet decomposition.

Time Series Classification: RCF classification is evaluated on 40 UCR datasets against recurrent, convolutional, fully connected, and wavelet-based baselines.Wavelet-RCF replaces mWDN with standard MDWD, isolating the value of trainable decomposition parameters.
Time Series Classification: FCN-RCF wins on 19 of 40 datasets and achieves the smallest MPCE among competitors.FCN itself wins 9 datasets with MPCE 0.023, while MLP-RCF and ResNet-RCF outperform their corresponding baselines on 37 and 27 datasets, respectively.
Time Series Classification: Trainable mWDN improves RCF over fixed wavelet decomposition, while wavelet frequency information remains beneficial for classification.ResNet-RCF generally outperforms Wavelet-RCF, indicating a tradeoff between wavelet priors and training-data likelihoods.
Time Series Forecasting: In Scenario I, mLSTM achieves the best prediction performance as period length varies from 5 to 30 minutes.Prediction error generally decreases for all models as period length increases, while mLSTM consistently outperforms wLSTM.
Time Series Forecasting: In Scenario II, mLSTM again outperforms wLSTM and other baselines as prediction intervals vary from 0 to 30 minutes.Unlike Scenario I, prediction errors generally increase with interval length because uncertainty increases.

5 INTERPRETATION

mWDN’s wavelet-derived middle layers retain interpretable frequency meanings, while importance analysis identifies which input elements and layers matter for different tasks and datasets.

Frequency interpretation: mWDN middle-layer outputs correspond to frequency components running from high to low across decomposition levels.This pattern appears in both mLSTM and RCF outputs.
Importance analysis: Importance analysis quantifies the contribution of each middle layer to an mWDN-based model’s final output.The method is extended from input-element sensitivity to middle-layer importance.
WuxiCellPhone: In WuxiCellPhone forecasting, recent input elements are more important, and lower-frequency mWDN layers have higher importance.Only higher-importance layers also exhibit the temporal importance pattern of the input elements.
ECGFiveDays: In ECGFiveDays classification, the most important elements lie around timestamps 100–110, corresponding to the electrocardiogram T-Wave.The text associates this region with heart relaxation and preparation for the next contraction.
ECGFiveDays: High-frequency layers are more important for ECGFiveDays because abnormal fluctuations distinguish heart-disease cases, unlike similar low-frequency trends.The authors present this as evidence that mWDN supports interpretability and exploration of deep-learning black-box behavior.

6 RELATED WORKS

Related work covers time-series classification, forecasting, and frequency analysis, emphasizing the contrast between learned temporal models and separately applied signal transforms.

Time Series Classification: Time-series classification methods include distance-based, feature-based, and ensemble approaches, with deep learning contributing automatic representative feature learning.Distance-based methods include Euclidean distance and DTW, while traditional feature-based methods often use handcrafted features.
Time Series Forecasting: Time-series forecasting predicts future values from past and present data and includes classical ARIMA variants such as ARIMAX and SARIMA.The supplied passage frames forecasting as broadly used across application domains.
Frequency Analysis: Frequency analysis uses transforms such as discrete wavelet, Fourier, and Z-transforms, but existing TSC/TSF applications usually apply them independently during preprocessing.The passage contrasts this preprocessing use with interaction between transforms and model training.

7 CONCLUSIONS

The paper develops trainable wavelet-based models for frequency-aware time-series classification and forecasting, then evaluates their performance and interpretability.

Contributions: mWDN embeds wavelet frequency learning into deep-learning frameworks while keeping all parameters trainable.The paper further builds separate mWDN-based models for classification and forecasting.
Contributions: RCF and mLSTM target time-series classification and forecasting, respectively, and experiments report superiority to state-of-the-art competitors.The evaluation uses abundant real-world datasets, according to the conclusion.
Interpretability: An importance-analysis method identifies important time-series factors and supports the interpretability merit of mWDN.The conclusion presents this as an additional contribution alongside the frequency-aware models.

APPENDIX

The appendix describes derivative-based sensitivity calculations through convolutional, LSTM, and fully connected layers for importance analysis.

Derivative propagation: The model’s derivative with respect to middle-layer outputs is computed layer by layer using the chain rule.The appendix applies this calculation to the proposed network components.
Layer types: The proposed models contain convolutional, LSTM, and fully connected layers, whose derivative calculations are treated separately.The appendix explicitly introduces these layer types before discussing their derivatives.
Convolutional layers: For convolutional layers, the derivative uses the one-dimensional convolution structure, kernel elements, and activation-function derivative.The convolutional output is described as an L × 1 × C matrix connected through a k × 1 × C kernel.
LSTM layers: For LSTM layers, the appendix expresses derivatives through the recurrent unit output and its history state stored in the memory cell.The supplied text introduces the output at time t and the associated intermediate quantity.
Fully connected layers: Fully connected layers use their output expression to obtain the corresponding partial derivative.The supplied passages provide the layer-specific derivative setup without stating a further interpretation.