Human Activity Recognition from Wearable Sensor Data Using Self-Attention

1 Introduction

• HAR applications include recognizing nurse-care activities and assessing rehabilitation or athletic exercises, where a system may assist health professionals in real time.The paper contrasts this assistance with assessment usually performed by an expert physiotherapist.Source: §1 · p. 1
• HAR remains challenging because sensor modalities, noisy signals, and spatial and temporal features vary across subjects and repeated performances.Complex activities such as stairup/down, running/jogging, watching TV, and ironing remain challenging for existing techniques.Source: §1 · p. 1
• The method adapts self-attention and positional encoding for sensor samples and adds a global attention layer to learn context across a time window.A fully connected layer classifies the resulting representation.Source: §1 · p. 2
• Across four datasets, the model outperforms DeepConvLSTM and ConvAE on benchmark test cases and shows stronger cross-subject generalization in LOSO-CV.The reported comparison covers sample-wise and window-wise benchmark tests.Source: §1 · p. 2
• The proposed non-recurrent architecture combines self-attention, sensor modality attention, and global temporal attention to build classification features.The attention layers are intended to capture spatio-temporal context in sensor signals.Source: §1 · p. 2
• The authors also analyze window-size effects and construct sensor-level attention maps described as intuitively explainable.The maps are intended to show the interpretability of the architecture's modules.Source: §1 · p. 2

2 Related Works

• Earlier wearable HAR used hand-crafted statistical or distribution-based features, while end-to-end deep learning learned representations alongside classifier optimization.The cited approaches include statistical features, wavelet or Fourier transformations, and CNN or recurrent combinations.Source: §2 · p. 2
• CNN–RNN hybrids and attention-augmented recurrent models were used to capture temporal characteristics and relationships among sensor modalities.Prior attention mechanisms included temporal attention, modality attention, or both, built on recurrent architectures.Source: §2 · p. 2
• The related-work review identifies an open architectural gap: no cited model combines self-attention for spatial feature-sequence context with temporal-context capture.Existing attention approaches discussed include attention-augmented LSTM, GRU, and ConvNet architectures.Source: §2 · p. 2

3 Proposed Self-attention Model

• The model foregoes recurrent networks and combines sensor modality attention, self-attention blocks, and global temporal attention for classification.Figure 1 depicts the architecture incorporating self-attention and global temporal attention.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Sensor Modality Attention

• Sensor modality attention uses 2-D convolution across time steps and sensor values to learn each sensor's contribution and weight the input.The resulting feature maps can be plotted to support model interpretability.Source: §3.1 · p. 3

3.2 Self Attention Block

• Self-attention compares each time step with the others, then uses the relative weights to incorporate relevant information into its representation.Distinct attention-head outputs are concatenated; position-wise feed-forward layers operate independently at each position, with residual connections and layer normalization.Source: §3.2 · p. 3

3.3 Global Temporal Attention

• Global temporal attention ranks time-step representations by importance and forms a weighted average as the feature vector for classification.The learned parameters include a hidden representation for each self-attention output and a parameter that captures temporal context in the attention score.Source: §3.3 · pp. 3–4
• Dropout is used in self-attention blocks, fully connected layers, and after positional encoding for regularization.Source: §3.3 · p. 4

4 Dataset Description

• Four benchmark datasets were used; Daphnet was excluded because it targets Parkinson’s-related freezing-of-gait recognition.The selected datasets were PAMAP2, USC-HAD, Opportunity, and SKODA.Source: §4 · p. 4
• PAMAP2 uses wrist, chest, and ankle IMUs and provides 18 activity classes across nine subjects; experiments used accelerometer and gyroscope readings.The dataset is sampled at 100 Hz.Source: §4 · p. 4
• Opportunity includes body-worn and ambient sensors, with experiments focused on 17 mid-level gestures and a null class exceeding 75% of data.It also includes five high-level activity classes and 13 low-level actions.Source: §4 · p. 4
• USC-HAD provides 12 activity classes from a hip-mounted motion node, while SKODA tracks assembly-line workers using accelerometers at 10 arm positions.USC-HAD’s authors note that its sensor placement and activity orientations challenge feature representation and segmentation.Source: §4 · p. 4

5 Experiment Setup

• The experiment setup covers preprocessing, model architectures, evaluation procedures, and performance measures.These components define the experimental comparisons reported in the paper.Source: §5 · p. 4

5.1 Preprocessing

• Datasets were downsampled to about 30 Hz for alignment, then represented with sliding windows whose labels use majority voting and whose overlap is tuned.Window size and overlap matter because activity duration and complexity vary; the activity window treats time steps and heterogeneous sensors as two dimensions.Source: §5.1 · p. 4

5.2 Implementation of Existing Architectures

• ConvAE classifies from a bottleneck representation, while DeepConvLSTM applies four temporal convolution layers followed by two LSTM layers.The comparison also includes attention-based HAR architectures, whose descriptions are cited from their respective papers.Source: §5.2 · pp. 4–5

5.3 Training and Test Procedures

• Sample-wise testing assigns each window’s output to its final time step; window-wise training uses 50%-overlapping windows labeled by their most frequent activity.The proposed model used two self-attention blocks, except USC-HAD used three; d was 128, and Adam used a learning rate of 0.001.Source: §5.3 · p. 5

5.4 Evaluation Metric

• Performance is compared using macro-averaged F1, giving each class equal weight regardless of its number of instances.Per-class F1 is calculated from precision and recall.Source: §5.4 · p. 5

6 Results

• Window-wise macro F1 reached 0.67 versus 0.58 (DeepConvLSTM) and 0.60 (ConvAE) on Opportunity; the model beat compared attention models on all datasets except Opportunity.The benchmark comparison also reports better USC-HAD scores: 0.50 sample-wise and 0.55 window-wise, versus DeepConvLSTM’s 0.42 and 0.38 and ConvAE’s 0.42 and 0.46.Source: Table 2 · p. 6, §6 · p. 6
• In LOSO-CV, macro F1-scores were consistently higher than DeepConvLSTM and ConvAE in sample-wise and window-wise tests; Skoda was excluded because it has one subject.The reported scores average across held-out subjects.Source: Table 3 · p. 6, §6 · p. 6
• Sample-wise scores improved over both baselines on PAMAP2 and USC-HAD, but fell to 0.61 from 0.66 (DeepConvLSTM) on Opportunity and were slightly lower on Skoda.The authors note significant improvement on Opportunity in window-based tests, consistent with the model being designed for window-based output.Source: §6 · p. 6
• For PAMAP2 LOSO-CV subject 102, F1-scores were 0.93 sample-wise and 0.98 window-wise, versus DeepConvLSTM’s 0.47 and 0.33 and ConvAE’s 0.35 in both cases.For subject 108, the model scored 0.79 sample-wise and 0.88 window-wise, versus DeepConvLSTM’s 0.27 and 0.28 and ConvAE’s 0.40 and 0.47, respectively.Source: §6 · p. 6
• The proposed model was less sensitive to window-size changes than other models; datasets with complex activities required relatively longer windows.Figure 3 presents performance against different window sizes.Source: §6 · p. 6, Figure 3 · p. 7
• PAMAP2 attention maps assign different sensor placements greater weight by activity, with the distributions described as intuitively explainable.Ironing and vacuum cleaning emphasize hand sensors; ascending stairs emphasizes hand and ankle sensors, and walking has a similar distribution to ascending stairs.Source: §6 · p. 7, Figure 4 · p. 7

• The proposed HAR architecture adapts the Transformer and uses attention mechanisms instead of recurrent layers to generate classification features.Source: §7 · pp. 7–8
• On benchmark test subjects, the model outperforms existing state-of-the-art methods on all datasets except Opportunity for sample-wise classification, and beats DeepConvLSTM and ConvAE window-wise.Source: §7 · p. 8
• Experiments cover PAMAP2, Opportunity, Skoda, and USC-HAD using leave-one-subject-out evaluation and both sample-wise and window-wise classification.Source: §7 · p. 8
• The experiments omit window-wise comparisons with newly published models, a comparison the authors plan to extend in future work.Source: §7 · p. 8