Source-linked AI summary
Understanding and Improving Recurrent Networks for Human Activity Recognition by Continuous Attention
Ming Zeng, Haoxiang Gao, Tong Yu, Ole J. Mengshoel, Helge Langseth, Ian Lane, Xiaobing Liu
TL;DR
Recurrent HAR models can encode irrelevant signals and sensor modalities while remaining difficult to interpret. The paper introduces temporal and sensor attention with continuity constraints, and reports improved mean F1 scores, attention-based interpretability, and state-of-the-art results across three datasets.
Problem
LSTM-based HAR may retain irrelevant signal components, mishandle unequal sensor importance, and provide limited insight into model decisions.
Method
The paper combines temporal attention, sensor attention, and continuity regularization for recurrent HAR models.
Results
The proposed approaches improve mean F1 scores compared with the state of the art and improve interpretability through attention visualization.
Takeaways & Limitations
Attention-based models focus on salient signal components and important sensor modalities while making recurrent HAR behavior more understandable.
Takeaways & Limitations
Quantifying each timestep’s contribution remains difficult because LSTM information accumulates over time, and attention can mismatch continuous attention in some cases.
Abstract
from arXiv · showhide
Deep neural networks, including recurrent networks, have been successfully applied to human activity recognition. Unfortunately, the final representation learned by recurrent networks might encode some noise (irrelevant signal components, unimportant sensor modalities, etc.). Besides, it is difficult to interpret the recurrent networks to gain insight into the models' behavior. To address these issues, we propose two attention models for human activity recognition: temporal attention and sensor attention. These two mechanisms adaptively focus on important signals and sensor modalities. To further improve the understandability and mean F1 score, we add continuity constraints, considering that continuous sensor signals are more robust than discrete ones. We evaluate the approaches on three datasets and obtain state-of-the-art results. Furthermore, qualitative analysis shows that the attention learned by the models agree well with human intuition.
INTRODUCTION
The paper addresses noisy and difficult-to-interpret recurrent models for human activity recognition by using temporal and sensor attention, plus continuity constraints. These mechanisms focus on salient signal components and sensor modalities while supporting model interpretation.
- Motivation: LSTM-based HAR can be affected by irrelevant signal components, unequal sensor importance, and limited interpretability.FOG-related abnormalities may occur only briefly, while accelerometers, gyroscopes, magnetometers, and sensor locations can contribute unequally.
- Proposed attention mechanisms: Temporal attention highlights important parts of time-series signals through the LSTM hidden layer.It is designed to ignore unimportant input segments and emphasize salient ones.
- Proposed attention mechanisms: Sensor attention reweights sensor modalities at the LSTM input layer according to their importance during training.This mechanism addresses differences among sensor modalities in activity recognition.
- Continuous attention: Continuous attention constraints regularize temporal and sensor attention to favor continuity in sensor signals.The constraints are introduced because continuous signals are considered more robust than discrete attention patterns.
- Interpretability: Visualizing attention weights provides insight into which input regions the models use and improves interpretability.The paper presents visualization as a way to understand recurrent-model behavior.
RELATED WORK
The paper situates its approach among hand-crafted features, deep feature-learning methods, CNN-based sensor fusion, and recurrent models used for HAR. It frames HAR as sequence classification with LSTM as a core building block.
- Feature extraction: Traditional HAR commonly uses hand-crafted statistical or transform-based features derived from raw sensor signals.Examples include mean, variance, entropy, correlation coefficients, Fourier transforms, and wavelet transforms.
- Deep feature learning: Deep neural networks reduce reliance on manually designed features by learning representations from data.Prior work applied DNNs, DBNs, CNNs, and related models to HAR, with varying treatment of temporal order and sensor information.
- CNN-based methods: CNN-based HAR methods can fuse multiple sensor modalities by representing signals as image-like structures.In this representation, signal length forms the width and the number of sensors forms the height.
- Recurrent models: LSTMs are recurrent models used to capture long-term information in time-series HAR data.Prior work combined LSTMs with convolutional layers or ensembles to improve recognition performance and robustness.
- Problem formulation: The paper formulates HAR as sequence classification, mapping sensor-reading sequences to activity labels with LSTM as a basic classifier component.Each time step contains a D-dimensional vector of sensor-channel readings.
Standard LSTM
The standard LSTM uses four gates to encode an input sequence into hidden states, with the final hidden state representing the preceding sequence for classification. Its memory mechanism helps handle long-range dependencies, but irrelevant or noisy signals can still complicate learning.
- The LSTM is a recurrent network with input, forget, output, and cell activation components.
- At the final timestep, h_T encodes previous sequence information and feeds a softmax classifier for the target class.
- The encoder can represent the whole sequence in a fixed-length hidden representation for classification.
- Standard LSTM memory cells mitigate long-distance dependency problems, but noisy or irrelevant signals remain difficult to handle reliably.
LSTM with Temporal Attention
Temporal attention replaces reliance on only the final hidden state with a weighted combination of hidden states from all timesteps. Learned attention weights let the model revisit earlier information and emphasize more important parts of the signal.
- Temporal attention retains hidden vectors for every timestep and references them at the final classification step.
- The representation H is computed as a weighted sum of previous hidden states rather than using only h_T.
- Attention weights α_t are learned from the sequence, whereas standard LSTM fixes α_T to 1 and earlier weights to 0.
- A bilinear score with learnable parameter W_α computes attention so the model can revisit information and focus on important signal parts.
LSTM with Sensor Attention
Sensor attention models the varying importance of sensor modalities by reweighting the input signal before subsequent processing. Its weights depend on both prior attention history and the current input.
- Sensor attention is applied at the input layer to fuse sensor modalities according to their varying importance.
- Each sensor-attention weight uses previous attention history β_{t−1} and the current input signal x_t.
- The reweighted signal input is formed through element-wise multiplication before classification.
- Temporal attention uses scalar α_t weights for hidden representations, while sensor attention uses vector β_t weights across channels.
- In the standard LSTM, sensor weights can be viewed as a uniform distribution.
Improving Attention LSTM with Continuous Attention
The paper adds continuity regularization because activity signals are continuous, whereas sharply switching attention can select disconnected points. The combined objective penalizes discontinuous temporal and sensor attention while optimizing recognition loss.
- Attention distributions can be overly sharp for sensor signals, whose informative evidence should form consecutive series rather than disconnected points.
- Continuous temporal attention regularization encourages attention weights to vary continuously across timesteps.
- Continuous sensor-attention regularization discourages switching between sensor modalities back and forth.
- The method focuses on sensor-modality attention because finer channel-level granularity requires more training data.
- The final loss combines recognition loss with temporal and sensor continuity regularizers.
EXPERIMENTS
The experiments used three public HAR datasets and a fixed LSTM training configuration on a Tesla K20c GPU server.
- Three public HAR datasets were evaluated using machine learning experiments on a Tesla K20c GPU server with 64G memory.The datasets reflect different activities, contexts, and sensor configurations.
- The LSTM configuration used one layer with a 128-dimensional hidden representation, ADMA optimization, a 0.05 learning rate, and gradient normalization at 1.
Datasets and Setup
The study evaluates attention-based HAR models on PAMAP2, Daphnet Gait, and Skoda using mean F1 and dataset-specific train, validation, and test protocols.
- Datasets: PAMAP2 contains 12 activities from 9 participants using multimodal recordings from hand, chest, and ankle IMUs.The data include 52 dimensions, a 5.12-second sliding window with 78% overlap, and approximately 473k samples after downsampling.
- Datasets: Daphnet Gait records accelerometer data from ankle, knee, and trunk sensors for binary detection of freezing-of-gait incidents in 10 Parkinson’s disease patients.
- Datasets: Skoda records 46 assembly-line activities with 19 accelerometers, while experiments recognize right-arm gestures using 10 accelerometers.The processed data contain 60 dimensions and approximately 190k samples.
- Evaluation: Performance is evaluated with mean F1, weighting class-specific F1 scores according to their sample proportions.The supplied formula defines class precision and recall using true positives, false positives, and false negatives.
Comparing with Traditional RNN Models for HAR
Continuous temporal attention improves recognition over standard LSTM baselines while directing attention toward temporally informative activity signals. Across examples, the learned weights emphasize activity-bearing portions of windows and can downweight irrelevant or static segments.
- Baselines: The evaluation compares single attention-based models with standard LSTM, DeepConvLSTM, and LSTM-S baselines.The study focuses on single-model comparisons rather than ensemble evaluation.
- Recognition results: Around 14.50%, 17.00%, and 3.40% mean F1 improvements were achieved on the three datasets versus the LSTM baseline.Continuous Temporal Attention also outperformed LSTM-S on DG and Skoda.
- Recognition results: Around 1.8% and 7.7% mean F1 improvements over the state-of-the-art were obtained on PAMAP2 and DG by combining continuous sensor and temporal attention.The same passage reports a negative impact on Skoda, potentially because its activities depend on sensor interaction rather than a single sensor.
- Statistical comparison: The reported LSTM with Continuous Temporal Attention is significantly better than the LSTM baseline with p-value < 0.01.
- Temporal attention: On a walking window where only the beginning contains walking, continuous temporal attention emphasizes hidden vectors around timesteps 0–43 and decays after approximately timestep 70.Temporal attention can look back to the end of the walking signal, whereas standard LSTM uses the final hidden vector.
- Temporal attention: For a window containing walking throughout, temporal attention focuses near the end, consistent with later recurrent states being more informative.Continuous temporal attention attends toward the end consecutively in this example.
- Temporal attention: Continuous temporal attention assigns less attention to timesteps 20–40, corresponding to a flat ankle-sensor signal.
- Temporal attention: For a running window whose activity appears only in the last half, continuous temporal attention increasingly focuses on hidden vectors from timestep 80 onward.The visualization compares standard LSTM, temporal attention, and continuous temporal attention weights.
Visualizing Important Sensor Modalities
Sensor attention differentiates modality importance rather than assigning equal weights to all sensors. In the walking example, both sensor-attention variants emphasize ankle and arm signals.
- Sensor attention: Unlike the attention models, LSTM without sensor attention treats all sensor modalities equally.Figure 8 compares LSTM without sensor attention, sensor attention, and continuous sensor attention for walking on PAMAP2.
- Sensor attention: Sensor attention and continuous sensor attention place high emphasis on ankle Acc2 and gyroscope signals and arm Acc2 signals.
DISCUSSION
The discussion identifies unresolved issues in interpreting temporal attention and modeling sensor interactions, while concluding that the proposed attention-based approaches improve HAR performance and interpretability.
- Dealing with sensor interaction: The sensor attention model currently infers each attention state from the corresponding raw signal and the previous attention state.Future work proposes another LSTM to model sensor-attention state transitions and capture more complicated sensor interactions.
- Better understanding for temporal attention: LSTM state accumulation makes it difficult to determine how much each timestep contributes to classification.The paper suggests gradient analysis during backpropagation to quantitatively evaluate the impact of previous signals.
- Better understanding for temporal attention: Attention and continuous attention mismatch in some cases, motivating further analysis of the models’ behavior.Figure 5 is cited as an example of this mismatch.
- Conclusion: The proposed approaches improve mean F1 score over the state of the art and improve interpretability through visualization results.The models focus on salient signal components and important sensor modalities.