Source-linked AI summary
Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables
Nils Y. Hammerla, Shane Halloran, Thomas Ploetz
TL;DR
HAR research lacked systematic evidence about which deep models and parameter settings suit different wearable-sensor tasks. This paper compares deep, convolutional, and recurrent approaches across three datasets and thousands of experiments, finding task-dependent model differences and strong recurrent performance, including state-of-the-art results on Opportunity.
Problem
HAR commonly relies on hand-crafted features, while prior deep-learning studies provide limited systematic evidence about model suitability, parameter exploration, and reproducibility across tasks.
Method
The paper compares deep, convolutional, and recurrent models across three HAR datasets using randomly sampled configurations, detailed recurrent training procedures, and a novel recurrent regularization approach.
Results
Bi-directional LSTMs outperform the current state-of-the-art on Opportunity, while recurrent models perform especially well for short, ordered activities and CNNs are recommended for prolonged repetitive activities.
Takeaways & Limitations
Model choice should reflect activity structure, and practitioners should consider parameter-exploration behavior rather than peak performance alone.
Takeaways & Limitations
The context of an individual movement sample is not well defined and may depend on movement type and wider behavioral context.
Abstract
from arXiv · showhide
Human activity recognition (HAR) in ubiquitous computing is beginning to adopt deep learning to substitute for well-established analysis techniques that rely on hand-crafted feature extraction and classification techniques. From these isolated applications of custom deep architectures it is, however, difficult to gain an overview of their suitability for problems ranging from the recognition of manipulative gestures to the segmentation and identification of physical activities like running or ascending stairs. In this paper we rigorously explore deep, convolutional, and recurrent approaches across three representative datasets that contain movement data captured with wearable sensors. We describe how to train recurrent approaches in this setting, introduce a novel regularisation approach, and illustrate how they outperform the state-of-the-art on a large benchmark dataset. Across thousands of recognition experiments with randomly sampled model configurations we investigate the suitability of each model for different tasks in HAR, explore the impact of hyperparameters using the fANOVA framework, and provide guidelines for the practitioner who wants to apply deep learning in their problem setting.
1 Introduction
HAR has relied largely on manually designed features and conventional classifiers, but deep learning may better address complex behaviors while making model selection and evaluation more systematic.
- HAR commonly uses sliding-window segmentation, hand-crafted feature extraction, and supervised classification on body-worn sensor data.
- Complex behaviors, including activities relevant to medical applications, challenge manually tuned HAR approaches.
- Deep learning can replace manually designed feature extraction, but practitioners lack clear guidance for selecting suitable architectures and parameters.
- The paper systematically explores deep, convolutional, and recurrent models across three HAR recognition problems using more than 4,000 experiments.
2 Deep Learning in Ubiquitous Computing
Wearable-sensor HAR analyzes high-resolution multivariate time series through segmentation and feature extraction, while deep models increasingly replace manual representations. Prior work has explored feed-forward, convolutional, and recurrent approaches, but recurrent models had not modeled sensor samples at the lowest level.
- Wearable-sensor movement data are multivariate time series with relatively high spatial and temporal resolution, commonly analyzed through segmentation and feature extraction.
- Deep belief networks and pre-trained fully connected networks were among the early attempts to replace manual feature selection, with mixed results for deep belief networks.
- Convolutional networks became the most popular deep-learning approach in ubicomp HAR and improved state-of-the-art performance in some specialized domains.
- Recurrent models are suited to temporal dependencies, but prior HAR work paired them with CNNs and used them to model temporal structure only at an abstract level.
- Before this work, recurrent networks had not modeled movement data at the lowest possible level: sequences of individual sensor samples.
3 Comparing deep learning for HAR
The paper compares deep, convolutional, and recurrent HAR models through broad random hyperparameter exploration, while adapting recurrent training to sample-level wearable-sensor sequences. It also distinguishes model architectures by their temporal context and application requirements.
- 3 Comparing deep learning for HAR: The study compares deep-learning approaches across representative HAR datasets by randomly sampling hyperparameter configurations and analyzing their performance across thousands of experiments.
- 3.1 Deep feed-forward networks (DNN): Deep feed-forward networks use up to five ReLU hidden layers followed by a softmax group, with dropout and max-in norm regularization.
- 3.2 Convolutional networks (CNN): CNNs retain the time-by-sensor matrix structure and use temporal convolution, pooling, fully connected layers, and dropout before softmax classification.
- 3.3 Recurrent networks: LSTM models recurrently process previous states, while bi-directional LSTMs combine information from both future and past context.
- 3.3 Recurrent networks: Forward LSTMs support real-time inference without future observations, whereas bi-directional LSTMs are suited to offline analysis.
- 3.3 Recurrent networks: The study evaluates frame-level LSTM-F, sample-level forward LSTM-S, and sample-level bi-directional b-LSTM-S settings.
- 3.4 Training RNNs for HAR: HAR sample context is not well defined and may depend on movement type and wider behavioral context, complicating recurrent training and window selection.
- 3.4 Training RNNs for HAR: RNN mini-batches preserve sample ordering by extracting sequences from randomly initialized positions, but they cannot stratify each mini-batch by class distribution.
4 Experiments
The experiments compare deep, convolutional, and recurrent models across three representative HAR datasets, using broad hyperparameter sampling and analyses designed to support reproducible model selection.
- Experimental design: More than 4,000 randomly sampled model configurations are evaluated across the benchmark datasets to compare model suitability and hyperparameter effects.The experiments sample configurations under approximately equal computation-time budgets and use fANOVA to estimate hyperparameter contributions.
- Datasets: Three datasets represent distinct HAR problems: short manipulative gestures, prolonged repetitive activities, and Parkinsonian freezing-of-gait detection.Opportunity covers kitchen gestures, PAMAP2 lifestyle and exercise activities, and Daphnet Gait a medical two-class recognition problem.
- Datasets: Opportunity uses 30 Hz recordings from 12 body locations and 18 gesture annotations collected from four participants.The dataset includes five runs per subject and excludes sensors with packet loss.
- Datasets: PAMAP2 records five sensor modalities from inertial units on the hand, chest, and ankle, producing 52-dimensional data across nine participants.The recordings cover 12 lifestyle activities over ten hours in total.
- Datasets: Daphnet Gait contains nine-dimensional accelerometer data from ten participants with Parkinson’s disease and uses one-second windows with 50% overlap.The task detects freezing-of-gait incidents for a future situated prompting system.
- Analysis and metrics: Hyperparameters are grouped into learning, regularisation, and architecture categories, with variability attributed to individual parameters and higher-order interactions.Mean f1-score is used because the datasets are highly class-distribution biased, while weighted f1-score supports comparison with prior Opportunity results.
5 Results
Recognition performance varies substantially across model families and hyperparameter settings, with recurrent approaches achieving strong results and dataset-dependent hyperparameter effects. The best-performing bidirectional LSTM surpasses prior state of the art on OPP, while recurrent-model performance depends on specific design choices.
- More than 15% mean f1-score separates b-LSTM-S from DNN on OPP, compared with 12% on DG and 7% on PAMAP2.
- 4% mean f1-score separates b-LSTM-S from the current state-of-the-art on OPP, while the best CNN exceeds earlier CNN results by more than 5% on both reported metrics.
- CNN configurations include failures but otherwise vary little, whereas DNNs show peak-to-median gaps of up to 35.7% on OPP.On PAMAP2, CNN peak-to-median performance differs by only 7% mean f1-score.
- Learning parameters have the most consistent effect in CNNs, while DNN effects vary by dataset and networks deeper than three hidden layers perform worse.Learning parameters matter most on PAMAP2, architecture matters most on OPP, and shallow networks outperform deeper variants in the reported experiments.
- LSTM-F performance on OPP depends critically on carry-over probability, with pcarry = 0.5 working well across most settings.Always retaining or always forgetting the internal state produces low performance.
- For b-LSTM-S, the number of units per layer has a surprisingly large effect on performance and should be tuned early.
6 Discussion
The study evaluates deep, convolutional, and recurrent models for wearable-sensor HAR and examines their suitability across tasks and parameter settings. Bidirectional LSTMs outperform the current state of the art on the Opportunity benchmark, while model choice depends on activity characteristics and tuning priorities.
- Bidirectional LSTMs outperform the current state-of-the-art on Opportunity by a considerable margin.
- Recurrent networks perform significantly better than CNNs on short activities with a natural ordering, while CNNs are recommended for prolonged and repetitive activities.The paper attributes the recurrent advantage to contextualising observations across long periods and recommends CNNs because their average performance makes suitable configurations more likely to be found.
- Bi-directional RNN performance is most affected by the number of units per layer, whereas learning rates should generally be explored before architecture.
- DNNs require substantial parameter exploration because poor preliminary performance can coexist with a large gap between peak and median recognition performance.The paper cautions practitioners not to discard DNNs after limited exploration.