Source-linked AI summary
Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline
Zhiguang Wang, Weizhong Yan, Tim Oates
TL;DR
Time series classification has often depended on preprocessing, feature engineering, or ensembles, motivating a simpler end-to-end baseline. The paper evaluates MLP, FCN, and ResNet models on benchmark datasets using raw time series. FCN achieves the best performance on three metrics at first sight, while FCN and ResNet belong to a statistically indistinguishable best group with COTE, MCNN, and BOSS; CAM identifies label-specific contributing regions.
Problem
Existing time-series classification approaches commonly require heavy preprocessing and feature engineering, while prior deep-learning evaluations provide limited benchmark evidence.
Method
The paper evaluates MLP, FCN, and ResNet baselines through pure end-to-end training on raw time series, using global average pooling in convolutional models.
Results
FCN indicates the best performance on three metrics, while FCN and ResNet are not significantly different from COTE, MCNN, and BOSS on MPCE.
Takeaways & Limitations
The models provide a simple baseline for time-series classification, and CAM identifies contributing raw-data regions for specific labels.
Abstract
from arXiv · showhide
We propose a simple but strong baseline for time series classification from scratch with deep neural networks. Our proposed baseline models are pure end-to-end without any heavy preprocessing on the raw data or feature crafting. The proposed Fully Convolutional Network (FCN) achieves premium performance to other state-of-the-art approaches and our exploration of the very deep neural networks with the ResNet structure is also competitive. The global average pooling in our convolutional model enables the exploitation of the Class Activation Map (CAM) to find out the contributing region in the raw data for the specific labels. Our models provides a simple choice for the real world application and a good starting point for the future research. An overall analysis is provided to discuss the generalization capability of our models, learned features, network structures and the classification semantics.
I. INTRODUCTION
Time series classification has traditionally relied on distance measures, feature engineering, or ensembles, while deep end-to-end approaches remain comparatively limited. The paper evaluates neural-network baselines on raw time series and uses CAM to identify label-specific contributing regions.
- Distance-based methods classify raw time series with predefined similarities such as Euclidean distance or DTW, with DTW plus k-nearest neighbors serving as a longstanding baseline.
- Feature-based methods extract global or local patterns, quantize them into representations such as Bag-of-Words, and classify the resulting features.
- Ensemble approaches combine multiple classifiers or feature sets, including Elastic Ensemble, Shapelet Ensemble, and COTE's 35-classifier ensemble.
- Existing approaches generally require substantial preprocessing and feature engineering, while prior CNN efforts have limited evaluations or use transformations such as sliding windows and downsampling.
- The paper evaluates MLP, FCN, and ResNet models on the same 44 benchmark datasets, finding FCN and ResNet comparable or better than COTE and MCNN through end-to-end raw-data training.
- Global average pooling enables CAM to identify regions in raw time series that contribute to specific predicted labels.
II. NETWORK ARCHITECTURES
The paper tests three neural-network architectures as a comprehensive baseline: a three-layer MLP, an FCN, and a deeper ResNet structure. The MLP uses ReLU and dropout to support deeper learning and generalization.
- Three deep neural-network architectures are tested to provide a comprehensive baseline: MLP, FCN, and ResNet.
- The baseline MLP stacks three fully connected layers of 500 neurons, followed by a softmax output layer.
- Dropout is applied at each layer's input to improve generalization, while ReLU supplies the nonlinear activation and helps prevent gradient saturation.
- The architecture differs from earlier MLPs mainly through ReLU and dropout; the authors use three layers as a fundamental baseline because excessive depth can leave many ReLU neurons inactive.
- Dropout rates for the input, hidden, and softmax layers are 0.1, 0.2, and 0.3, respectively.
B. Fully Convolutional Networks
The FCN processes time series with stacked one-dimensional convolutional blocks, batch normalization, ReLU, and global average pooling rather than pooling or a fully connected feature layer. The related ResNet extends this block design with shortcut connections and also uses global average pooling.
- The FCN acts as a feature extractor whose final output is produced by a softmax layer.
- Each FCN block contains a one-dimensional convolution, batch normalization, and ReLU, using kernels of sizes 8, 5, and 3 without striding.
- Three convolution blocks use 128, 256, and 128 filters, exclude pooling, and apply batch normalization to speed convergence and improve generalization.
- Global average pooling replaces a fully connected layer, substantially reducing the number of weights before the final softmax classification.
- ResNet adds shortcut connections to residual blocks, allowing gradients to flow directly through lower layers in very deep networks.
- The ResNet reuses the FCN convolutional blocks, stacks three residual blocks with filter counts 64, 128, and 128, then applies global average pooling and softmax.
III. EXPERIMENTS AND RESULTS
The experiments compare deep neural network baselines and established methods on 44 UCR datasets using error-based evaluation. MPCE is introduced to account for differing numbers of classes, while paired testing assesses whether model differences are significant.
- Experimental setup: The networks are evaluated on the same 44-dataset UCR subset, with z-normalization as the only preprocessing described.Normalization uses the training split’s mean and standard deviation for both training and test data.
- Evaluation protocol: The study reports test error rates and dataset-level best-performance counts, supplemented by average rankings for benchmark comparison.The comparison includes eight other benchmark methods.
- Evaluation protocol: The number of best-performing datasets and ranking statistics are biased because they depend on top performance, model pools, and omit the number of classes.These measures can therefore give different comparative impressions across datasets and model sets.
- Mean Per-Class Error: Mean Per-Class Error (MPCE) evaluates classification performance across multiple datasets while incorporating each dataset’s number of classes.Its intuition is the expected error rate for a single class across all datasets.
- Mean Per-Class Error: A paired T-test on per-class error identifies whether MPCE differences between models are statistically significant.The test is applied to PCE values across models.
C. Results and Analysis
The proposed deep models are compared with established baselines on 44 UCR datasets using multiple evaluation measures. FCN performs especially strongly, while FCN and ResNet join a statistically indistinguishable best group with several benchmark methods.
- Benchmark comparison: The benchmark includes TSBF, PROP, BOSSVS, SE1, COTE, MCNN, and 1NN-DTW, alongside the proposed neural networks.COTE combines weighted votes from 35 classifiers, while BOSSVS ensembles BOSS models with different window lengths.
- Benchmark comparison: FCN achieves the best performance on three of four reported metrics, while ResNet remains competitive on MPCE and rankings.The metrics are reported in Table I for the 44 UCR datasets.
- Statistical analysis: FCN, ResNet, COTE, MCNN, and BOSS form a best group because their MPCE differences are not significant in the paired T-test.The remaining approaches are grouped separately based on the MPCE test results.
- Model behavior: ResNet ranks third among the 11 models and tends to overfit more easily than FCN.The paper still places ResNet in the first group without a significant difference from four other best models.
- Benchmark comparison: FCN and ResNet classify raw time series from scratch without heavy preprocessing or feature crafting, achieving performance comparable to or better than COTE and MCNN.MCNN requires data transformation, downsampling, and window slicing.
- Model behavior: The three-layer MLP achieves results comparable to 1NN-DTW without a significant difference.The reported comparison attributes this performance to ReLU and dropout in the MLP.
IV. LOCALIZE THE CONTRIBUTING REGIONS WITH CLASS ACTIVATION MAP
Global average pooling enables class activation maps that localize time-series regions contributing to specific predicted labels. The maps can be upsampled to the input length and reveal label-specific discriminative patterns.
- CAM interpretation: FCN’s global average pooling provides a natural extension to class activation maps for interpreting class-specific regions in time-series data.CAMs visualize predicted class scores and highlight discriminative subsequences detected by the convolutional network.
- CAM construction: For each class, the CAM assigns temporal importance based on the final convolutional activations and the softmax weights associated with that class.The resulting map indicates which temporal locations contribute to classification.
- CAM construction: When the last convolutional output is shorter than the input, the CAM is upsampled to identify relevant regions at the original time-series resolution.This preserves localization in the raw input despite the network’s temporal downsampling.
- CAM interpretation: CAM examples highlight discriminative regions for the correct classes, with different categories emphasizing different contributing regions.The paper illustrates this behavior on two datasets.
- Dataset examples: On CBF, label 0 focuses on the sharp drop, label 1 on a sharp rise followed by a smooth decline, and label 2 on a middle plateau.The StarLightCurve analysis similarly distinguishes visually similar labels through different attention patterns.
A. Overfitting and Generalization
The study examines overfitting and generalization in deep time-series classifiers, while analyzing learned representations and filters. Despite expected overfitting on small UCR datasets, architectural choices support strong generalization and interpretable temporal features.
- Neural networks were expected to overfit substantially because UCR datasets are small and model selection used training loss without validation settings.
- Dropout improves MLP generalization, while batch normalization improves convolutional-network training speed and generalization.
- Global average pooling reduces parameters by replacing the fully connected layer, contributing to improved generalization.
- CAM localizes class-specific contributing regions in raw time series during the same forward pass used for classification.
- Feature Visualization and Analysis: GASF visualizes multi-scale temporal correlations; its diagonal contains the original values, while off-diagonal entries encode cosine summations across time points.
- Feature Visualization and Analysis: FCN and ResNet filters extract local temporal features and dependencies across intervals and frequencies, with deeper-layer filters remaining similar to preceding layers.
C. Deep and Shallow
The paper explores whether a very deep ResNet architecture can remain effective for time-series classification. Its shortcut connections support gradient flow, but greater depth increases overfitting risk and regularization demands.
- ResNet uses shortcut connections that let gradients flow directly through lower layers while learning complex patterns.
(a)MLP
The paper contrasts shallow and deep network behavior through visualized learned filters and test performance. ResNet remains competitive, but on these relatively simple UCR patterns it overfits more than FCN.
- Filter visualizations compare learned representations across MLP, FCN, and ResNet on the Adiac dataset.
- On UCR data, ResNet still tends to overfit, and its test performance is not as good as FCN because the patterns are comparatively simple.
D. Classification Semantics
The paper analyzes whether different classifiers learn similar feature spaces and decision criteria. PCA-based PCE analysis indicates that the proposed models complement existing approaches, while MLP behavior differs from convolutional models.
- The analysis asks whether models with similar classification performance learn convergent or divergent feature spaces and classifiers.
- PCA preserves large pairwise distances when reducing representation dimension for PCE analysis.
- The proposed models are comparatively distant from other benchmarks in PCE space, indicating complementary learned features and classification criteria.
- FCN and ResNet embeddings are close, whereas MLP embeddings form an isolated category with different classification behavior.
- PCE distributions are compared after PCA dimension reduction to examine relationships among approaches.
- The study presents end-to-end neural baselines without heavy preprocessing or feature crafting for time-series classification.