Source-linked AI summary
RTFN: A Robust Temporal Feature Network for Time Series Classification
Zhiwen Xiao, Xin Xu, Huanlai Xing, Shouxi Luo, Penglin Dai, Dawei Zhan
TL;DR
Time-series feature networks must represent both local patterns and relationships among features, while existing embedding structures may miss global variations in non-periodical data. The paper proposes RTFN, combining a temporal feature network with an LSTM-based attention relation network, and reports strong supervised and unsupervised performance across benchmark datasets.
Problem
Existing feature networks emphasize local features, while extracting sufficient representations and relationships among features remains challenging, particularly for global variations in non-periodical data.
Method
RTFN combines a temporal feature network for local-feature extraction with an LSTM-based attention network for relation extraction.
Results
RTFN achieves decent performance in supervised classification and unsupervised clustering, with its supervised algorithm best on 39 of 85 univariate datasets.
Takeaways & Limitations
RTFN provides a feature-extraction framework evaluated in both supervised and unsupervised time-series learning settings.
Takeaways & Limitations
Embedding attention and LSTM may not capture global variations in long univariate and multivariate non-periodical datasets, and transmission layers may lose features.
Abstract
from arXiv · showhide
Time series data usually contains local and global patterns. Most of the existing feature networks pay more attention to local features rather than the relationships among them. The latter is, however, also important yet more difficult to explore. To obtain sufficient representations by a feature network is still challenging. To this end, we propose a novel robust temporal feature network (RTFN) for feature extraction in time series classification, containing a temporal feature network (TFN) and an LSTM-based attention network (LSTMaN). TFN is a residual structure with multiple convolutional layers. It functions as a local-feature extraction network to mine sufficient local features from data. LSTMaN is composed of two identical layers, where attention and long short-term memory (LSTM) networks are hybridized. This network acts as a relation extraction network to discover the intrinsic relationships among the extracted features at different positions in sequential data. In experiments, we embed RTFN into a supervised structure as a feature extractor and into an unsupervised structure as an encoder, respectively. The results show that the RTFN-based structures achieve excellent supervised and unsupervised performance on a large number of UCR2018 and UEA2018 datasets.
1. Introduction
Time-series classification requires representations that capture both local patterns and relationships among extracted features. RTFN addresses this with separate local-feature and relation-extraction networks, achieving strong supervised and unsupervised results.
- Time-series applications require effective feature and relation extraction to use data fully for classification.
- Existing approaches include distance-based, feature-based, single-network, and dual-network models for mining patterns and representations.
- Dual-network models separate local-feature extraction from relation extraction, with the latter targeting hidden connections among extracted features.
- Cascaded attention–LSTM models can lose useful representations, while embedding models may miss global variations in long non-periodical series.
- RTFN combines a CNN-based residual temporal feature network for local features with an LSTM-based attention network for relation extraction.
- RTFN-based structures perform strongly in supervised and unsupervised settings across UCR2018 and UEA2018 datasets.
2. Related Work
Related work distinguishes single-network models from dual-network models and cascading from embedding approaches for hybridizing attention with LSTM. The paper motivates a dual-network design that embeds LSTM within attention to address global variations in non-periodical time series.
- Deep-learning methods for time-series classification are broadly organized into single-network and dual-network models.
- Single-network models jointly handle feature and relation extraction, including convolutional architectures such as InceptionTime and OS-CNN.
- Dual-network models use separate local-feature and relation-extraction networks, and reported comparisons indicate better classification performance than single-network models.
- Attention–LSTM hybrids are commonly implemented as cascading or embedding models, with embedding models compactly integrating the two networks.
- Existing hybrid models may not handle global variations of non-periodical time-series data well, motivating a dual-network algorithm with LSTM embedded in attention.
3. RTFN
RTFN combines TFN for local-feature extraction with LSTMaN for discovering relationships among sequential features. It is used in supervised classification and unsupervised clustering structures.
- Overview: RTFN combines a temporal feature network and an LSTM-based attention network to extract local features and relationships among them.TFN captures local features, while LSTMaN extracts intrinsic relationships and helps recover representations ignored by TFN.
- Temporal Feature Network: TFN uses Conv1D blocks, multi-head convolutional layers, and self-attention to obtain higher-level multi-scale local features and relate their positions.Two multi-head layers contain four Conv1D blocks each, with self-attention between them.
- LSTM-based Attention Network: LSTMaN contains two identical LSTM-based attention layers that extract increasingly complex relationships among features.The first layer extracts basic relationships, while the second extends them to more complex regularities.
- LSTM-based Attention Network: LSTMaN maps LSTM-derived queries, keys, and values to attention outputs through compatibility weighting and a weighted sum of values.The query, key, and value matrices carry long- and short-term features before attention produces its output.
- RTFN-based Structures: RTFN concatenates TFN local features with LSTMaN relationships, providing combined representations for downstream structures.The resulting representation is embedded into supervised and unsupervised configurations.
- RTFN-based Structures: The supervised structure adds dropout and a fully connected classifier, whereas the unsupervised structure uses an RTFN encoder, decoder, and K-means classifier.The unsupervised structure trains using reconstruction loss rather than a K-means loss.
4. Experiments and Analysis
The experiments describe the setup, metrics, ablation study, and evaluations of RTFN-based supervised and unsupervised structures.
- Experimental Analysis: The experimental section introduces the setup and performance metrics before presenting an ablation study.
- Experimental Analysis: RTFN-based supervised classification and unsupervised clustering are evaluated separately.
4.1. Experimental Setup
The experiments use selected UCR2018 and UEA2018 datasets for supervised classification and unsupervised clustering, with kernel-size studies guiding TFN configuration.
- Datasets: The study evaluates supervised classification and unsupervised clustering on standard univariate and multivariate time-series datasets.UCR2018 provides univariate datasets, while UEA2018 provides multivariate datasets.
- Datasets: The supervised evaluation selects 85 UCR2018 datasets, including 65 short-medium and 20 long datasets.A long dataset is defined as having a length over 500.
- Datasets: The multivariate archive UEA2018 contains 30 datasets spanning seven application areas.The listed areas include audio spectra, electrocardiogram, electroencephalogram, human activity recognition, motion, and magnetoencephalography.
4.2. Performance Metrics
The evaluation uses accuracy-based metrics for supervised classification and rand index for unsupervised clustering, together with win/tie/lose and average-rank summaries.
- Supervised Classification: Supervised algorithms are ranked using win, tie, lose, mean accuracy, and AVG rank based on top-1 accuracy.Win, tie, and lose count datasets where an algorithm performs better than, equivalently to, or worse than others.
- Supervised Classification: For each supervised algorithm, the best value sums its corresponding win and tie outcomes across datasets.
- Supervised Classification: AVG rank measures the average difference between a model’s accuracy and the best accuracy among all models.
- Unsupervised Clustering: Unsupervised clustering is evaluated with the rand index, whose average across datasets is reported as AVG RI.RI uses positive and negative time-series pair counts in the clustering and the dataset size.
4.3. Ablation Study
The ablation study examines TFN’s activation and self-attention components and compares alternative LSTM-based attention configurations. LeakReLU, self-attention, and two attention layers are supported by the reported comparisons.
- Component Design: RTFN combines TFN for local-feature extraction with LSTMaN for relation extraction.LSTMaN uses LSTM-based attention to model relationships among extracted features at different positions.
- TFN Components: TFN outperforms TFN w ReLU on every supervised-classification and unsupervised-clustering dataset in the ablation comparison.One reported top-1 accuracy comparison is 0.833333 for TFN versus 0.611111 for TFN w ReLU.
- TFN Components: TFN also outperforms TFN w/o SelAtt on all evaluated datasets, supporting the inclusion of self-attention.The paper attributes this to self-attention relating different time-series positions and enriching extracted features.
- LSTMaN Depth: TFN+2LSTMaL consistently outperforms TFN+1LSTMaL, while TFN+3LSTMaL is equivalent in almost all cases except Car.The paper explains that the second layer exposes relationship details that a third layer may merely transmit.
- LSTMaN Depth: TFN+2LSTMaL has lower model complexity than TFN+3LSTMaL, including CPU times of 32.421894s and 35.440211s on SemgHandGendeCh2.The paper presents two attention layers as a balance between accuracy and model complexity.
- Relation-Extraction Comparisons: TFN+2LSTMaL outperforms TFN+AttLSTM and TFN+LSTMTAG on every supervised and unsupervised comparison dataset.It also outperforms TFN alone, indicating complementary roles for TFN and LSTMaN.
4.4. Evaluation of the RTFN-based Supervised Structure
The RTFN-based supervised structure is evaluated against existing algorithms on univariate and multivariate UCR2018 and UEA2018 datasets. It achieves the strongest reported aggregate outcomes across these comparisons, including on long time series.
- Evaluation Scope: The supervised evaluation compares algorithms using win, lose, tie, MeanACC, and AVG rank on 85 univariate and 30 multivariate datasets.
- Multivariate Time Series: RTFN performs best in tie and second best in win on the reported multivariate comparison, yielding the first position in best.
- Univariate Time Series: On 20 long time-series datasets, RTFN is the best algorithm by obtaining the highest Best and MeanACC values.
- Multivariate Time Series: The paper attributes RTFN’s multivariate performance to TFN extracting features and LSTMaN relating representations across variables.LSTMaN is described as discovering relationships among representations associated with different variates.
4.5. Evaluation of the RTFN-based Unsupervised Clustering
The RTFN-based unsupervised clustering uses RTFN features with K-means and performs strongly across 36 selected UCR2018 datasets, ranking second overall and outperforming separate K-means on all but two datasets.
- Experimental setup: The evaluation compares RTFN-based K-means with state-of-the-art unsupervised algorithms on 36 selected UCR2018 datasets.RI, average RI, and average rank are used for comparison.
- Overall results: DTCR ranks first and the RTFN-based unsupervised clustering ranks second among the compared algorithms.This ranking is reported for all 36 datasets.
- Method: The method uses an auto-encoding structure to update parameters and K-means to classify features obtained by RTFN.The approach is described as simple in structure while achieving decent performance on the 36 selected datasets.
- Interpretation: The reported performance is attributed to RTFN extracting sufficient features, including features hidden deeply in the input data.The passage contrasts this feature extraction ability with ordinary feature extraction networks.
- Comparison with K-means: The RTFN-based method outperforms separate K-means on all but two datasets.The comparison uses RI values shown in Figure 6.
5. Conclusion
RTFN combines convolutional local-feature extraction with LSTM-based attention for relational feature extraction. It performs strongly in supervised classification and unsupervised clustering across UCR2018 and UEA2018 datasets, with potential for use in other learning frameworks.
- Architecture: TFN extracts local features, while the LSTM-based attention network discovers intrinsic relationships among learned representations.These components form the two principal feature-extraction roles in RTFN.
- Overall results: RTFN achieves decent performance in both supervised classification and unsupervised clustering.The conclusion presents this as the overall experimental finding.
- Supervised classification: The supervised RTFN structure performs best on 39 of 85 UCR2018 univariate datasets and 15 of 30 UEA2018 multivariate datasets.These comparisons use the latest reported results from the supervised classification community.
- Supervised classification: RTFN wins on 11 of 20 long representative single-network-based models.This result is reported within the comparison of supervised classification performance.
- Implication: The results indicate that RTFN has potential to be embedded in other learning frameworks for time series problems across various domains.This is stated as a conclusion from the experimental results.