PaperlayerView exact revision on arXiv

Source-linked AI summary

DanHAR: Dual Attention Network For Multimodal Human Activity Recognition Using Wearable Sensors

Wenbin Gao, Lei Zhang, Qi Teng, Jun He, Hao Wu

arXiv:2006.14435v4cs.CV

TL;DR

Multimodal HAR needs to capture informative sensor channels and temporal regions, but prior attention approaches either rely on recurrent backbones or focus temporally without channel information. DanHAR combines channel and temporal attention with CNN or residual-network backbones and evaluates it across public and weakly labeled datasets. It achieves reported improvements over CNN and residual-network baselines while maintaining small parameter overhead, and visualization shows which sensors and sequence portions receive priority.

  • Problem

    Multimodal HAR lacks a unified attention approach that preserves CNN representation power while modeling both channel and temporal dependencies.

  • Method

    DanHAR blends channel and temporal attention with CNN or residual-network backbones for multimodal sensor signals.

  • Results

    DanHAR performs best among compared algorithms across reported datasets, with accuracy improvements over CNN and residual-network baselines and small parameter overhead.

  • Takeaways & Limitations

    Attention visualizations indicate that DanHAR prioritizes different sensors and important portions of long sequences, improving sensor-signal comprehensibility.

Abstract

from arXiv · show

Human activity recognition (HAR) in ubiquitous computing has been beginning to incorporate attention into the context of deep neural networks (DNNs), in which the rich sensing data from multimodal sensors such as accelerometer and gyroscope is used to infer human activities. Recently, two attention methods are proposed via combining with Gated Recurrent Units (GRU) and Long Short-Term Memory (LSTM) network, which can capture the dependencies of sensing signals in both spatial and temporal domains simultaneously. However, recurrent networks often have a weak feature representing power compared with convolutional neural networks (CNNs). On the other hand, two attention, i.e., hard attention and soft attention, are applied in temporal domains via combining with CNN, which pay more attention to the target activity from a long sequence. However, they can only tell where to focus and miss channel information, which plays an important role in deciding what to focus. As a result, they fail to address the spatial-temporal dependencies of multimodal sensing signals, compared with attention-based GRU or LSTM. In the paper, we propose a novel dual attention method called DanHAR, which introduces the framework of blending channel attention and temporal attention on a CNN, demonstrating superiority in improving the comprehensibility for multimodal HAR. Extensive experiments on four public HAR datasets and weakly labeled dataset show that DanHAR achieves state-of-the-art performance with negligible overhead of parameters. Furthermore, visualizing analysis is provided to show that our attention can amplifies more important sensor modalities and timesteps during classification, which agrees well with human common intuition.

I. INTRODUCTION

The introduction frames multimodal wearable-sensor HAR as a difficult time-series classification and labeling problem. DanHAR addresses limitations of existing attention approaches by combining channel and temporal attention on CNN-based models, with evaluation across public and weakly labeled datasets.

  • Multimodal HAR uses sliding windows of wearable-sensor signals to classify activities for applications including sports, gaming, health care, and monitoring.
  • Deep HAR improves representation and avoids hand-crafted features, but collecting accurately annotated sensor sequences is expensive and laborious.Sensor recordings from accelerometers and gyroscopes are harder to interpret and label than many other modalities.
  • Coarse-grained labels could reduce manual labeling if models infer activity types and locate target activities within long sensor sequences.
  • Existing attention methods combine spatial-temporal modeling with GRU or LSTM, whereas CNN-based hard and soft attention focus temporally but omit channel information.
  • DanHAR blends channel and temporal attention on CNNs, while experiments span four public HAR datasets and a weakly labeled dataset.The paper also reports ablations and visualization analyses of attention weights.

II. RELATED WORKS

Related work establishes attention as a useful mechanism for selectively emphasizing informative features. Prior multimodal HAR methods use automatic deep feature extraction and temporal or sensor attention, motivating channel-wise modeling.

  • Attention selectively focuses processing on salient parts and has improved tasks including image classification, object detection, and semantic segmentation.
  • Channel attention has been incorporated into convolutional architectures through methods such as second-order pooling, CBAM, scSE, GC-Net, and DAN.
  • Deep learning has become a dominant multimodal HAR approach because it learns feature representations directly from raw sensor readings.
  • Prior HAR attention models use LSTM backbones with temporal and sensor attention to highlight important portions of sensor time series.

III. MODEL

DanHAR combines convolution, channel attention, and temporal attention to model multimodal sensor signals. Its network processes fixed-length windows and sequentially learns which feature channels and temporal regions matter.

  • DanHAR addresses multimodal HAR by combining channel and temporal attention, with CNN or residual-network backbones for temporal-spatial sensor patterns.
  • The model uses fixed-length sliding windows, convolutional layers, a channel-attention submodule, and a temporal-attention submodule.Adjacent windows may overlap to preserve activity continuity.
  • Sequential channel and temporal attention learns which sensor features and timesteps to emphasize during classification.

A. Channel Attention Submodule

The channel-attention submodule derives channel weights from pooled feature-map information. Average and max pooling provide complementary channel descriptors, which are transformed through reduced fully connected layers and a sigmoid.

  • For feature map A ∈ R^C×H×W, channel attention uses global average pooling and max pooling over temporal and sensor dimensions.C denotes channels, while W and H denote temporal and sensor dimensions.
  • The pooled descriptors pass through two fully connected layers with nonlinear interaction to generate channel weights.
  • A sigmoid function converts the transformed pooled features into channel-attention weights.
  • The fully connected layers use a reduction ratio r, with weight sizes C × (C/r) and (C/r) × C to reduce parameters.

B. Temporal Attention Submodule

The temporal attention submodule aggregates channel information with average- and max-pooling, then combines pooled features to form temporal attention. It is combined with channel attention through element-wise multiplication.

  • Temporal attention aggregates channel information using average-pooling and max-pooling operations.
  • Four aggregation strategies are considered: max-out, multiplication, addition, and concatenation.
  • The temporal-attention convolution uses a 7×1 filter, with average- and max-pooling features represented by g1 and g2.
  • Concatenation is selected to aggregate channel-index information because it avoids information loss compared with addition.
  • The temporal and channel attention maps are multiplied element-wise, with each map scaled to [0,1].

C. Residual Network

DanHAR uses a residual network backbone with three convolutional building blocks. Each block contains two convolutional layers and shortcut connections that combine identity and residual mappings.

  • The residual backbone contains three sets of convolutional layers, each organized as a building block with two convolutional layers.
  • Shortcut connections skip each block for identity mapping and add the block’s residual mapping to form the final underlying mapping.
  • The six-layer residual network uses double-layer skips with ReLU and batch normalization between layers.

IV. EXPERIMENTS AND RESULTS

Experiments evaluate DanHAR on four public HAR benchmark datasets, using residual and standard CNN backbones to assess the dual attention mechanism under comparable architectures.

  • The supervised experiments use WISDM, UNIMIB SHAR, PAMAP2, and OPPORTUNITY benchmark datasets.
  • The datasets represent different contexts and use either multiple sensor nodes or smartphones worn by participants.
  • The dual attention mechanism is incorporated into feed-forward architectures through end-to-end training.
  • For fair comparison, the mechanism is embedded in both residual networks and standard CNNs without residual skip connections.

A. Experiment Results and Performance Comparison

DanHAR combines channel and temporal attention with CNN or residual-network backbones and is evaluated across public, multimodal, and weakly labeled HAR datasets. Across the reported experiments, it generally achieves lower test error and higher accuracy than the corresponding baselines, with negligible parameter overhead.

  • Model evaluation: DanHAR combines dual channel and temporal attention with CNN and residual-network backbones for multimodal HAR.The experiments use standard CNN and residual network models, with attention modules trained end to end.
  • WISDM: 0.73% and 0.53% accuracy improvements over CNN and residual-network baselines are reported on WISDM, with a 0.62% gain over the best published result.The reported dual-attention result is described as best reported on WISDM, with negligible parameter overhead.
  • UNIMIB SHAR: 1.24% and 1.91% accuracy gains over the two baselines are reported on UNIMIB SHAR, while attention adds no significant parameter increase.DanHAR also exceeds CNN with local loss by 0.96% and another dual-residual method by 2.99%.
  • OPPORTUNITY: 2.89% and 0.7% performance improvements over standard CNN and residual network are reported on OPPORTUNITY, with about a 1.75% accuracy gain over the best published result.DanHAR is reported as best among the compared algorithms and exceeds DeepConvLSTM by 3.85% accuracy.
  • Weakly labeled data: 3.76% and 3.93% accuracy improvements over the two baselines are reported on the weakly labeled dataset, with a 4.82% margin over DeepConvLSTM.The authors attribute the weakly supervised improvement to the role of channel-wise attention.

B. The Ablation studies

The ablation studies evaluate how dual attention arrangement, attention visualization, and classification errors contribute to understanding DanHAR. Sequential channel-temporal attention performs best, while visualizations identify important time segments and sensor modalities.

  • Experimental design: The ablations compare sequential channel-temporal, sequential temporal-channel, channel-only, and temporal-only attention under otherwise identical experimental settings.Experiments were conducted on WISDM, PAMAP2, and the weakly labeled dataset.
  • Attention arrangement: Sequential channel-temporal attention improves performance over temporal-channel, channel-only, and temporal-only arrangements.The reported improvements are 0.23%, 0.73%, and 0.76%, respectively.
  • Attention visualization: Channel attention indicates which sensor modalities matter for different activities, while temporal attention identifies important signal components in weakly labeled sequences.The visual analyses use PAMAP2 for multimodal channel weights and weakly labeled data for temporal weights.
  • Classification errors: DanHAR reduces PAMAP2 misclassifications from 93 for the residual network to 75 activities.Confusions remain between similar classes such as rope jumping and walking, whose signal vibrations are similar.

V. CONCLUSION

DanHAR combines channel and temporal attention with CNN or residual-network backbones to address multimodal HAR. Across public and weakly labeled datasets, it improves performance with small parameter overhead and provides interpretable attention visualizations.

  • Interpretability: Temporal attention visualizations show that the proposed modules can locate target activities in weakly labeled sensor sequences.The weakly labeled examples include jogging, going downstairs, and jumping amid background activity.
  • Interpretability: Channel-attention visualizations examine how different sensor modalities contribute to activities in PAMAP2.The analysis uses IMUs placed at different positions on the human body.
  • Evaluation: PAMAP2 confusion matrices compare DanHAR with a residual network.The comparison focuses on classification behavior across activity classes.
  • Proposed method: DanHAR combines channel and temporal attention with CNN or residual-network backbones to capture multimodal spatial-temporal patterns.The method is designed to improve both network understanding and multimodal HAR performance.
  • Evaluation: DanHAR is evaluated on four public HAR datasets and a weakly labeled dataset, achieving significant performance improvement with small parameter overhead.The public datasets are WISDM, PAMAP2, UNIMIB SHAR, and OPPORTUNITY.
Loading 2006.14435v4…