Source-linked AI summary
Deep ConvLSTM with self-attention for human activity decoding using wearables
Satya P. Singh, Aimé Lay-Ekuakille, Deepak Gangwar, Madan Kumar Sharma, Sukrit Gupta
TL;DR
Wearable-sensor activity decoding needs methods that avoid the limits of handcrafted features while capturing temporal and cross-sensor information. The paper combines CNN and LSTM representations with self-attention, then evaluates the approach across sampling strategies and six public datasets. The proposed model improves accuracy over a ConvLSTM baseline and previous methods, with statistically significant gains over prior state-of-the-art results except on WISDM.
Problem
Handcrafted wearable-sensor features are data-dependent and time-consuming, while existing CNN/LSTM approaches do not focus on latent relationships among multiple sensor time-series embeddings.
Method
The architecture combines CNN embeddings, LSTM temporal encoding, and self-attention that learns relationships among time points before activity classification.
Results
The proposed approach improves accuracy over ConvLSTM and previous state-of-the-art methods across tested datasets, with statistically significant gains over prior state-of-the-art results except WISDM.
Takeaways & Limitations
Self-attention provides a viable way to improve human-activity decoding from multiple wearable sensors across different sampling schemes.
Abstract
from arXiv · showhide
Decoding human activity accurately from wearable sensors can aid in applications related to healthcare and context awareness. The present approaches in this domain use recurrent and/or convolutional models to capture the spatio-temporal features from time-series data from multiple sensors. We propose a deep neural network architecture that not only captures the spatio-temporal features of multiple sensor time-series data but also selects, learns important time points by utilizing a self-attention mechanism. We show the validity of the proposed approach across different data sampling strategies on six public datasets and demonstrate that the self-attention mechanism gave a significant improvement in performance over deep networks using a combination of recurrent and convolution networks. We also show that the proposed approach gave a statistically significant performance enhancement over previous state-of-the-art methods for the tested datasets. The proposed methods open avenues for better decoding of human activity from multiple body sensors over extended periods of time. The code implementation for the proposed model is available at https://github.com/isukrit/encodingHumanActivity.
I. INTRODUCTION
Wearable-sensor activity decoding avoids some practical difficulties of camera-based monitoring, but handcrafted features are data-dependent and limited. The proposed CNN–LSTM architecture adds self-attention to focus on relationships among sensor time points, improving accuracy across six benchmark datasets.
- Wearable sensors provide portable measurements from body- or pocket-placed devices for monitoring health, living conditions, and daily activities.They combine input sensors, signal transfer, and signal-processing units.
- Handcrafted statistical and frequency-domain features are data-dependent, time-consuming to extract, and limited in number across datasets.
- Deep neural networks can learn features directly from raw inputs, reducing the need for handcrafted feature extraction.
- CNNs and LSTMs capture spatio-temporal information, but prior approaches did not focus on latent relationships among embeddings from multiple sensor time series.
- Self-attention learns relationships among input variables and significantly improves accuracy when combined with CNN and LSTM models across six benchmark datasets.
II. Proposed model for selecting sensors and time
The proposed model processes a fixed window of multivariate wearable-sensor time series through convolutional embeddings before temporal encoding and classification. Its modules learn local contextual features from sensor inputs using one-dimensional spatial convolutions.
- The model takes a time window of length T containing measurements from N wearable sensors.
- Embedding layer: Its embedding layer uses multiple one-dimensional convolution filters to learn local contextual or abstract features from sensor inputs.
- The architecture combines convolutional embeddings, one or more LSTM layers for temporal dynamics, self-attention over time points, and a SoftMax classification layer.
- Embedding layer: For filter k, the convolution output h_c,k(t) is formed from sensor input x(t) using filter weights w_k.
B. LSTM Encoder
The LSTM encoder models temporal dependencies in the convolution-derived sensor representations. Its output at each time point is a hidden representation whose size is determined by the encoder’s LSTM units.
- The encoder uses one or more LSTM layers to learn temporal information and dependencies from sensor time-series data.
- For one LSTM layer, the encoder output h_e(t) has dimension 1×E, where E is the number of hidden units.
C. Self-Attention Layer
The self-attention layer receives the sequence of LSTM encoder representations and learns weights over time points. These weights capture relationships in the sensor time series that support state-label prediction.
- Self-attention receives the concatenated encoder outputs h_e across the T time points after CNN and LSTM feature extraction.
- The attention module learns weight coefficients that capture relationships between time points in each sensor time-series sample.
- The attention score S has dimensions F×E and is computed from the encoder representation using learned attention weight matrices.D is the attention length and F is the output length.
- The coefficients α lie between 0 and 1 and assign different weights to encoder embeddings h_e(t) for each time point.
D. SoftMax Layer
The flattened attention score vector is passed to a softmax layer, which produces output-state probabilities for classification. Training minimizes cross-entropy cost using the Adam optimizer.
- The flattened attention score vector s is used as input to the output softmax layer.
- The softmax layer maps the attention representation to output states using a weight matrix W and bias vector b.C denotes the number of possible output states.
- The predicted state label is the output neuron receiving the maximum softmax value.
- The proposed model is trained by minimizing cross-entropy cost J with the Adam optimizer.
III. EXPERIMENTAL SETUP
The experimental setup validates the proposed and comparable approaches using six benchmark wearable-sensor datasets with varied subjects, activities, sensor configurations, and sampling rates. Dataset use involved appropriate permissions, consent, and repository approvals.
- Six benchmark wearable-sensor datasets were used to experiment with and validate the proposed and existing approaches.The datasets include MHEALTH, UTD-MHAD, USC-HAD, WHARF, WISDM, and Wireless Sensor Data.
- MHEALTH recorded body signals and vital signs from sensors on the left ankle, chest, and right wrist of 12 volunteers at 50 Hz.
- UTD-MHAD combines depth and wireless inertial data for 27 actions performed by 8 subjects at a 50 Hz inertial-sensor sampling rate.
- USC-HAD contains 12 daily activities from 14 subjects using a motion node on the right hip, with a maximum sampling rate of 100 Hz.
- WHARF uses a wrist-worn 3-axis accelerometer to record 14 activities from 17 subjects at a 30 Hz sampling rate.
- Human-subject dataset collection followed permissions, consent, and approvals documented in the respective repositories.
B. Sample generation process
Raw multichannel time series are divided into fixed-size temporal windows, while trial-level separation prevents signals from one data group from mixing with another during cross-validation. SNOW is used to increase sample counts, despite its overlap.
- Raw sensor time series are split into equal fixed-size temporal windows, omitting an incomplete final sample.
- SNOW applies a fixed-size window with 50% overlap between successive windows, which can introduce bias.
- FNOW removes overlap between successive windows but produces fewer samples for deep-learning requirements.
- The study uses LOTO-style trial separation during 10-fold cross-validation so raw activity signals are not mixed across training, validation, and testing groups.
- SNOW is retained to generate the high number of samples required by the deep-learning models.
C. Evaluation protocols
Evaluation uses accuracy, recall, and F1-score, with metric interpretation depending partly on class balance and error-cost symmetry. The experiments compare sampling strategies and the proposed method against a deep ConvLSTM baseline across six datasets.
- The study evaluates classifiers using accuracy, recall, and F1-score.
- Accuracy is recommended for balanced data, whereas F1-score is used when data are imbalanced.
- Table II reports proposed-approach accuracy as a percentage with standard deviation for different sample-generation methods.
- Table III compares the proposed approach with deep ConvLSTM using accuracy, recall, and F1-score across six benchmark datasets.
- Accuracy is more appropriate when false positives and false negatives have similar costs.
IV. Results and discussions
The experiments compare the proposed self-attention architecture with ConvLSTM under multiple sampling and cross-validation settings. Results favor the proposed model overall, while performance depends on training-set size and sampling overlap.
- Experimental settings: Three CNN filters performed best across all datasets during hyperparameter tuning.The experiments varied CNN filters from 1 to 12 and LSTM units from 8 to 64.
- Sampling strategies: LOTO was selected for subsequent experiments because it avoids overlap between data samples.SNOW and FNOW could mix related training samples into testing data, whereas LOSO evaluates unseen subjects and showed high fold variance.
- Comparison with baseline: The proposed model outperformed ConvLSTM in accuracy and F1-score across datasets, except when the number of trials was relatively low.With small training sets, including USC-MHAD2 and MHEALTH, performance was similar to the baseline.
- Comparison with baseline: The proposed approach showed significant overall improvement over the ConvLSTM baseline.The comparison used multiple folds under the LOTO scheme and evaluated both accuracy and F1-score.
C. Comparative analysis with existing work
The study compares the proposed architecture with handcrafted-feature methods, ensemble learning, CNN, and ConvLSTM using adapted implementations and accuracy results. The proposed method achieved the strongest accuracy overall, with statistically significant gains over prior state-of-the-art methods except on WISDM.
- Comparison setup: The comparison covers handcrafted features, ensemble learning, CNN, and ConvLSTM methods using dataset-specific experimental settings.Direct comparison is difficult because prior studies vary in hyperparameters, datasets, and data-generation techniques.
- Handcrafted-feature baselines: The handcrafted-feature baseline uses five statistical features and a multilayer perceptron classifier.The features include mean, standard deviation, average resultant acceleration, and time between peaks.
- Handcrafted-feature baselines: The ensemble baseline combines J48, MLP, and logistic regression using soft voting.The statistical features are generated as in the earlier handcrafted-feature baseline.
- Neural baselines: The CNN comparison uses three convolutional layers, with smaller kernels for WHARF and WISDM because their sampling rate is low.The standard kernels are replaced with sizes (3, 2), (3, 1), and (3, 1) for those datasets.
- Results: The proposed method was superior in accuracy to all compared literature methods, with statistically significant gains on every dataset except WISDM.MLP with handcrafted features had the lowest performance, while ensemble learning and 2D CNN showed no significant difference overall.
V. CONCLUSIONS
The paper concludes that combining CNN and RNN spatio-temporal encoding with self-attention improves wearable-sensor activity recognition. Across several datasets and sampling schemes, the architecture improves accuracy over CNN/RNN baselines and prior state-of-the-art methods.
- Proposed architecture: The proposed architecture combines CNN spatial encoding, RNN temporal encoding, and self-attention over sensor-time-series embeddings.The self-attention layer generates feature representations from embeddings produced by the CNN and RNN.
- Evaluation: The architecture achieved comparable performance across different sample-generation schemes on five named wearable-sensor datasets.The conclusion names MHEALTH, USC-HAD, UTD-MHAD, WHARF, and WISDM.
- Conclusion: The proposed approach improved activity-recognition accuracy over CNN/RNN baselines and previous state-of-the-art approaches.The conclusion presents this comparison as evidence for the architecture’s effectiveness in human activity recognition.