Source-linked AI summary
On Attention Models for Human Activity Recognition
Vishvak S Murahari, Thomas Ploetz
TL;DR
Fixed temporal contexts may not fit activities with varying durations, motivating a data-driven way to identify relevant history. The paper adds attention layers to DeepConvLSTM, learns weights over past sensor context, and evaluates the result on benchmark HAR datasets, finding significant gains on some benchmarks and interpretable context patterns.
Problem
Fixed-size temporal contexts may be poorly suited to activities whose durations vary within and across classes, leaving the relevant history for classification uncertain.
Method
The paper adds an attention layer to DeepConvLSTM that learns weights over historical sensor representations and uses their weighted context for classification.
Results
Attention significantly improves performance on Opportunity and PAMAP2, while producing only marginal improvements on Skoda.
Takeaways & Limitations
Learned attention weights can focus HAR models on relevant history and provide additional insight into model behavior.
Abstract
from arXiv · showhide
Most approaches that model time-series data in human activity recognition based on body-worn sensing (HAR) use a fixed size temporal context to represent different activities. This might, however, not be apt for sets of activities with individ- ually varying durations. We introduce attention models into HAR research as a data driven approach for exploring relevant temporal context. Attention models learn a set of weights over input data, which we leverage to weight the temporal context being considered to model each sensor reading. We construct attention models for HAR by adding attention layers to a state- of-the-art deep learning HAR model (DeepConvLSTM) and evaluate our approach on benchmark datasets achieving sig- nificant increase in performance. Finally, we visualize the learned weights to better understand what constitutes relevant temporal context.
INTRODUCTION
HAR traditionally uses fixed sliding windows, but activity durations vary and the relevant temporal context is not known in advance. This paper explores attention models that learn relevant context weights and reports improved benchmark performance.
- INTRODUCTION: Fixed sliding windows make temporal-context choices difficult and may be unsuitable for activities with varying durations.Window decisions affect recognition and errors are difficult to recover from.
- INTRODUCTION: The paper asks whether models can automatically learn which historical sensor readings are relevant for representing activities.The authors also seek to externalize these data-driven context decisions for insight into the analyzed data.
- INTRODUCTION: Attention models learn weights over historical sensor readings, emphasizing context relevant to a classification decision.The weights represent the relative importance of input representations for supervised HAR.
- INTRODUCTION: Adding attention to a state-of-the-art deep-learning HAR model significantly increases performance on standard benchmarks.The evaluation uses Opportunity, PAMAP2, and Skoda.
BACKGROUND
HAR deep-learning research has emphasized CNNs and recurrent models, including DeepConvLSTM. CNNs rely on fixed-window representations, while recurrent models provide sequence modeling through internal memory.
- BACKGROUND: CNNs learn hierarchical representations through stacked filters and pooling, but analyze sequences using fixed sliding windows.The window converts time-series data into a fixed two-dimensional representation for convolution.
- BACKGROUND: LSTM-based recurrent models maintain internal memory through gated cells that feed back outputs and track hidden state.These models have been applied successfully in challenging HAR scenarios.
- BACKGROUND: DeepConvLSTM combines CNN representation learning with LSTM sequence learning and is described as a promising state-of-the-art HAR model.This combination motivates extending DeepConvLSTM with attention.
ATTENTION FOR HAR
Fixed temporal contexts can be poorly matched to the substantial duration variation within and across activity classes. The proposed approach uses attention to determine relevant context in a data-driven way while preserving the existing HAR architecture as the starting point.
- ATTENTION FOR HAR: Fixed-size temporal contexts may not yield ideal classification when activity durations vary substantially within and across classes.Previous deep-learning HAR methods represent every sensor reading with the same context size.
- ATTENTION FOR HAR: Attention models are explored as a way to automatically determine the temporal context relevant for modeling each activity.The intended effect is to adapt analysis windows in a data-driven manner.
- ATTENTION FOR HAR: The model begins with a large sliding-window context and adds an attention layer that rescales samples according to their relevance.Other hyperparameters are left unchanged to isolate the effect of introducing attention.
DeepConvLSTM and Attention
The attention model extends DeepConvLSTM by weighting recurrent hidden states representing parts of an input frame. A weighted historical context is added to the current state, and the resulting embedding is used for prediction.
- DeepConvLSTM and Attention: DeepConvLSTM processes a one-second Opportunity window through four convolution layers, pooling, and a two-layer LSTM.The input contains 24 samples and is converted into a time-by-feature representation before recurrent modeling.
- DeepConvLSTM and Attention: Attention analyzes the eight LSTM hidden states, treating the first seven as historical context and learning one weight for each.The eighth hidden state is treated as the current representation.
- DeepConvLSTM and Attention: The final attention-based embedding replaces the original model’s final hidden state for classification.The addition of the last hidden state acts as a skip connection from the recurrent layers.
- DeepConvLSTM and Attention: The model transforms historical states, scores them, applies softmax weights, and forms a weighted sum for the past-context embedding.The past-context embedding is added to the last hidden state to produce the final input-frame embedding.
EXPERIMENTS
The experiments evaluate attention-based HAR models on three diverse benchmark datasets using standard sample-wise protocols and report mean F1 scores. Training uses sliding-window frames, shuffled samples, and tuned regularization across all models.
- Datasets: The evaluation uses Opportunity, PAMAP2, and Skoda as diverse benchmarks for human activity recognition.The datasets differ in activity types and relative activity distributions.
- Evaluation: Recognition results are reported as mean F1 scores because all three datasets have imbalanced class distributions.Statistical significance is assessed with Wilson score intervals at 95% confidence.
- Preprocessing: All experiments extract processing frames with a sliding-window procedure initialized at one second of data.The supplied implementation passage does not specify the resulting frame length beyond this initialization.
- Preprocessing: Frames overlap by 50%, are randomly shuffled during training, and produce sample-wise rather than frame-wise predictions.These choices are used consistently across the studied models.
RESULTS
Attention models improve recognition over non-attention approaches on Opportunity and PAMAP2, while producing only marginal gains on Skoda. The results are reported as mean F1 scores under class imbalance, with significance assessed using Wilson intervals.
- Benchmark results: Attention models significantly improve performance over the state of the art on Opportunity and PAMAP2.The comparison uses mean F1 scores and Wilson score intervals with 95% confidence.
- Benchmark results: Skoda shows only marginal improvement when attention is introduced.The paper relates this pattern to similarly limited gains in other model evaluations on the dataset.
- Benchmark results: The authors conclude that Skoda has reached a performance level with little apparent potential for further improvement.This conclusion is specific to the reported benchmark evaluations.
DISCUSSION
The learned attention weights reveal how the model distributes temporal importance across hidden states and activities. They emphasize later states while retaining earlier context, suppressing the first two states, and spreading most evenly for “Open Door 3.”
- Attention-weight visualization: Figure 3 summarizes median attention weights across the first seven LSTM hidden states for each activity in the Opportunity test set.The weights represent relative importance learned by the best attention model.
- Attention-weight visualization: Most attention concentrates on the last few hidden states, while earlier states also receive weight instead of relying only on the final state.The paper attributes the later-state emphasis to their richer recurrent summaries and the earlier weighting to information present at the input frame’s start.
- Attention-weight visualization: The first two hidden states receive weights close to zero across all analyzed activities.The paper links this pattern to their short and relatively uninformative histories.
- Activity-specific patterns: “Open Door 3” has the most evenly spread weights across hidden states among the Opportunity activities.The paper connects this distribution to multiple smaller activities within the drawer-opening action and identifies it as a basis for future segmentation schemes.