Source-linked AI summary
The UEA multivariate time series classification archive, 2018
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, Eamonn Keogh
TL;DR
Multivariate time series classification lacked a broad, standardized benchmark, with algorithms evaluated on few datasets and without statistical comparisons. The paper constructs the first UEA MTSC archive: 30 standardized datasets with train/test splits, yielding benchmark performance that ranges from at least 99% accuracy on five datasets to no better than 50% on five others. The archive is an initial work in progress with planned expansion and limited ECG coverage.
Problem
Multivariate time series classification was being evaluated on very few datasets, and improvement claims were not based on statistical comparisons.
Method
The authors construct the first UEA MTSC archive as a collaborative 30-dataset collection with equal-length, complete series and train/test splits.
Results
Five datasets reach at least 99% accuracy with one or more benchmark classifiers, while five cannot exceed 50% accuracy.
Takeaways & Limitations
The archive provides a public basis for broader MTSC evaluation across datasets with widely varying classification difficulty.
Takeaways & Limitations
The archive is a first work-in-progress attempt, with limited ECG data and plans for expansion.
Abstract
from arXiv · showhide
In 2002, the UCR time series classification archive was first released with sixteen datasets. It gradually expanded, until 2015 when it increased in size from 45 datasets to 85 datasets. In October 2018 more datasets were added, bringing the total to 128. The new archive contains a wide range of problems, including variable length series, but it still only contains univariate time series classification problems. One of the motivations for introducing the archive was to encourage researchers to perform a more rigorous evaluation of newly proposed time series classification (TSC) algorithms. It has worked: most recent research into TSC uses all 85 datasets to evaluate algorithmic advances. Research into multivariate time series classification, where more than one series are associated with each class label, is in a position where univariate TSC research was a decade ago. Algorithms are evaluated using very few datasets and claims of improvement are not based on statistical comparisons. We aim to address this problem by forming the first iteration of the MTSC archive, to be hosted at the website www.timeseriesclassification.com. Like the univariate archive, this formulation was a collaborative effort between researchers at the University of East Anglia (UEA) and the University of California, Riverside (UCR). The 2018 vintage consists of 30 datasets with a wide range of cases, dimensions and series lengths. For this first iteration of the archive we format all data to be of equal length, include no series with missing data and provide train/test splits.
1 Introduction
The paper introduces the first iteration of the UEA multivariate time series classification archive, a collaborative 30-dataset collection designed to support more rigorous MTSC evaluation. The archive standardizes data formats and provides train/test splits while documenting storage and manipulation procedures.
- The first MTSC archive iteration contains 30 datasets spanning varied numbers of cases, dimensions, and series lengths.
- All datasets are formatted with equal-length series, no missing data, and supplied train/test splits.
- Most archive datasets had not previously been used for time series classification.
- The archive distributes multivariate problems in Weka multi-instance format, with separate files by dimension available for most datasets.Very high-dimensional datasets are exempted from separate per-dimension files to avoid substantially enlarging the archive.
- In the format, a relational attribute stores multiple series per case, while the header specifies series length and the data specify the number of series.The example defines series of length 8 and two series per case.
- The archive applies minimal preprocessing, retains donor-provided train/test splits, and draws data from UCI, brain-computer-interface, Kaggle, and researcher-created sources.
2 Human Activity Recognition
Human Activity Recognition classifies activities from accelerometer and/or gyroscope coordinates, and the archive includes nine such problems. BasicMotions and Cricket illustrate activity or motion data represented by multivariate sensor measurements and labeled cases.
- Human Activity Recognition predicts an activity class from accelerometer and/or gyroscope data.The HAR problems use either three or six coordinate dimensions.
- The archive includes nine Human Activity Recognition problems, making HAR its largest application group.
- BasicMotions: BasicMotions data were collected from four activities using a smartwatch’s 3D accelerometer and 3D gyroscope.The activities are standing, walking, running, and playing badminton, recorded at 10 Hz for ten seconds across five repetitions.
- BasicMotions: The BasicMotion example labels its first training case as Standing.
- Cricket: Cricket records twelve umpire signals performed ten times by four umpires using two wrist accelerometers.Each accelerometer measures three synchronous axes, producing a six-dimensional classification problem.
- Cricket: The Cricket example labels its first training case as Cancel Ball (1).
2.3 Epilepsy
The supplied passages describe several activity-recognition datasets, including sensor-based activity recordings and example training cases. They also document truncation and participant-based train/test splitting for one activity dataset.
- Epilepsy: The activity data were collected from six participants using a tri-axial accelerometer during four simulated activities.The activities were walking, running, sawing, and seizure mimicking.
- Epilepsy: Series were truncated to approximately 13 seconds after removing flat prefixes and suffixes, with random intervals selected from longer recordings.One incorrectly collected case was removed before final preparation.
- Epilepsy: The resulting dataset contains 275 cases, split by participants into three training participants and three testing participants.Participant identifiers were removed for consistency with the archive.
- Epilepsy: The archive figures show example first training cases for Epilepsy, ERing, and Handwriting, together with their class labels.The displayed labels are Epilepsy, Fist (2), and U (21).
2.6 Libras
The Libras dataset represents Brazilian sign-language hand movements as normalized two-dimensional trajectories. Its preprocessing samples each video uniformly and maps each trajectory into coordinate features.
- 2.6 Libras: LIBRAS contains 15 hand-movement classes with 24 instances per class, recorded from four people across two sessions.Each video corresponds to one movement and lasts about seven seconds.
- 2.6 Libras: Each movement is time-normalized to 45 frames, with segmented-hand centroids forming a discrete two-dimensional curve.The curves are normalized in unitary space.
- 2.6 Libras: The preprocessing maps each curve into a 90-feature representation of movement coordinates.The representation derives from the 45 sampled two-dimensional points.
- 2.6 Libras: An example first training case is shown for the Libras problem, with class label 1.The movement trajectory can be plotted as 45 ordered points.
2.8 RacketSports
The RacketSports problem classifies badminton and squash actions using smartwatch motion data. The archive also includes gesture and activity examples from related motion datasets.
- 2.8 RacketSports: Smartwatch recordings capture gyroscope and accelerometer x, y, z coordinates while students perform badminton or squash strokes.Data were sampled at 10 Hz over three seconds.
- 2.8 RacketSports: The classification task identifies both the sport and the stroke being performed.The recorded strokes are forehand and backhand in squash, and clear and smash in badminton.
- 2.8 RacketSports: The archive describes a separate gesture dataset containing eight simple accelerometer-generated gestures with three coordinate dimensions.Each series has length 315.
- 2.8 RacketSports: Figures provide example first training cases for RacketSports and UWaveGestureLibrary, including their class labels.The RacketSports example is labelled Badminton Smash, while the UWaveGestureLibrary example is labelled 1.
3 Motion Classification
The Motion classification group covers articulator, character, worm, and handwritten-digit trajectories. These datasets encode movement through multivariate sensor, pen, or shape-related time series and are formatted to common archive constraints.
- 3.1 ArticularyWordRecognition: ArticularyWordRecognition uses nine dimensions from electromagnetic articulograph sensors tracking speech-related tongue, lip, jaw, and head-independent movement.The original collection used twelve sensors with x, y, and z positions sampled at 200 Hz.
- 3 Motion Classification: The motion examples include first training cases for ArticularyWordRecognition, CharacterTrajectories, EigenWorms, and PenDigits.Their displayed labels are 1.0, g, wild-type (1), and 8, respectively.
- 3.2 CharacterTrajectories: CharacterTrajectories represents each instance as a three-dimensional pen-tip velocity trajectory for one of 20 character classes.All series were truncated to the shortest length, 182 observations, to conform with the repository.
- 3.3 EigenWorms: EigenWorms classifies individual Caenorhabditis elegans as wild-type or mutant using six time series of amplitudes projected onto six eigenworms.The archive contains 259 cases, with each truncated series having 17,984 observations.
- 3.4 PenDigits: PenDigits is a handwritten-digit task in which instances contain two-dimensional pen-tip coordinates and labels for digits 0 through 9.Spatial resampling produces eight points per instance.
4 ECG Classification
The archive includes ECG classification problems, but the authors found surprisingly few suitable datasets in this domain. The included problems use different signal representations and activity or cardiac labels.
- ECG classification is identified as an obvious application for multivariate time series classification, yet suitable datasets were surprisingly difficult to find.The authors note that PhysioNet data often requires bespoke processing and is not always an obvious classification problem.
- ECG datasets: AtrialFibrillation uses two-channel ECG recordings from 5-second atrial-fibrillation segments sampled at 128 samples per second.The channels are represented as the two dimensions of each multivariate case, with class labels including n and s.
- Examples: The archive illustrates these problems with first training cases labeled n for AtrialFibrillation and standing for StandWalkJump.
- ECG datasets: StandWalkJump represents spectrogram frequency bands as dimensions for classifying standing, walking, and jumping activities.Each class contains 9 instances, and the original recordings were sampled at 500 Hz before spectrogram construction.
5 EEG/MEG Classification
EEG and MEG problems cover applications in medicine, psychology, and human–computer interaction, with many datasets originating from brain–computer-interface competitions. The archive spans imagined movement, self-regulation, face detection, and hand-movement direction tasks.
- EEG and MEG classification has applications in medicine, psychology, and human–computer interaction, and most of the archive’s data came from BCI competitions.
- Examples: The section provides example first training cases for Finger Movements, SelfRegulationSCP1, SelfRegulationSCP2, FaceDetection, and HandMovementDirection.The examples use labels such as left, finger, negativity, 0, and 1 - right.
- EEG datasets: Finger Movements contains EEG recordings with 64 dimensions of length 3000 and labels for imagined finger or tongue movements.The train and test sets contain 278 and 100 cases, respectively; the best submitted solution achieved 91% test accuracy.
- EEG datasets: SelfRegulationSCP1 contains six-channel, length-896 trials labeled negativity or positivity, with 268 training trials and 293 test trials.The best approach reported an 11.3% test error rate.
- EEG datasets: SelfRegulationSCP2 contains seven-dimensional series of length 1152, while its test set has 180 trials and an unclear classification signal.The reported best error is 45.5%.
- MEG datasets: Other included problems cover MEG-based face-versus-scramble classification and four-way hand-movement direction classification.The hand-direction dataset preserves the original competition’s train/test split after merging trials from two subjects.
6 Audio Spectra Classification
The archive converts several audio classification problems into multivariate time series by treating spectrogram frequency bins as series over successive windows. The included datasets span bird sounds, heart sounds, insect wingbeats, phonemes, and Japanese vowels.
- Audio classification is converted from a univariate signal problem into a multivariate representation by making each spectrogram frequency bin a series over windows.
- Spectral datasets: DuckDuckGeese uses five-second, spectrogram-transformed recordings from Xeno Canto, with recordings downsampled to 44100Hz before transformation.The dataset distinguishes bird classes, including Black-bellied Whistling Duck.
- Examples: The archive illustrates these datasets with first training cases for DuckDuckGeese, Heartbeat, InsectWingbeat, Phoneme, and JapaneseVowels.
- Spectral datasets: Heartbeat contains five-second spectrogram representations of heart sounds, with normal and abnormal classes containing 113 and 296 instances.
- Audio datasets: InsectWingbeat stores spectrogram-derived power spectra, while Phoneme represents segmented speech as multivariate spectrogram data.Both datasets apply the archive’s frequency-band representation to audio-derived signals.
- Audio datasets: JapaneseVowels uses 12-dimensional linear-prediction time series padded to length 29 to classify nine male speakers.The training set has 30 utterances per speaker, while the test distribution ranges from 24 to 88 instances per speaker.
7 Other Problems
The remaining archive problems cover spectroscopy, traffic occupancy, and astronomical light curves, using domain-specific multivariate representations. Their formulations include bottle-independent alcohol classification, weekday prediction, and a fixed-dimensional snapshot of astronomical data.
- Examples: The section includes example first training cases for EthanolConcentration, PEMS-SF, and LSST.
- EthanolConcentration: EthanolConcentration classifies four alcohol concentrations from repeated spectral readings of water-and-ethanol solutions in 44 whisky bottles.The data contain three repeat readings for each bottle and solution batch.
- EthanolConcentration: Bottle properties introduce noise and structural defects, so the task is to identify concentration regardless of the containing bottle.The archive provides leave-one-bottle-out folds so all examples from a bottle are reserved for testing.
- EthanolConcentration: The ethanol dataset’s aligned wavelength dimensions are intended mainly to support noise cancellation or correction, motivating comparison with single-reading classification.The passage also suggests interval methods may benefit from the wavelength structure.
- Other datasets: PEMS-SF contains 15 months of daily freeway occupancy data, with each day represented as a 963-dimensional time series and labeled by weekday.
- Other datasets: LSST data represent object brightness over time through photon-flux measurements in six astronomical passbands.The archive formulation is a snapshot created from the competition’s published training set, using 36 dimensions to avoid truncating most instances.
8 Benchmark Results
Initial benchmarking used three standard 1-NN classifiers on raw and independently normalized data, revealing substantial variation in dataset difficulty. The authors identify these results as preliminary and plan broader validation.
- Three 1-NN classifiers were evaluated with Euclidean, dimension-independent DTW, and dimension-dependent DTW on raw and normalized data.Results were reported in Table 3 and summarized with a critical difference diagram.
- Five datasets reached at least 99% accuracy with one or more benchmark classifiers, while five did not exceed 50%.
- The benchmark results span a wide range of performances across the archive’s datasets.
- The authors describe these results as an initial benchmarking attempt and plan to expand them with resampling and/or cross-validation experiments.
9 Conclusions
The archive is presented as a first multivariate archive that remains a work in progress. The authors anticipate expansion and invite additional data, validated results, and error reports.
- The archive is explicitly characterized as a first attempt and a work in progress.
- The authors hope to release an expanded version in 2019 and welcome donations of data.
- They invite peer-reviewed, reproducible classifier results and reports of errors in the data or descriptions.