PaperlayerView exact revision on arXiv

Source-linked AI summary

Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition

Xiangbo Shu, Jiawen Yang, Rui Yan, Yan Song

arXiv:2112.10992v2cs.CVstat.ML

TL;DR

Elderly activity recognition must distinguish individual actions and human-object interactions despite similar motions and subtle cues. The paper proposes ESE-FN, which uses modal- and channel-wise ESE attentions with a multimodal loss to fuse RGB and skeleton features. On ETRI-Activity3D, ESE-FN achieves state-of-the-art accuracy and is also comparable with other methods for normal action recognition.

  • Problem

    Elderly activity recognition is challenging because activities contain similar motions, local human-object interactions, and subtle movement differences requiring discriminative RGB and skeleton information.

  • Method

    ESE-FN attentively fuses RGB and skeleton features with modal- and channel-wise ESE attentions and uses a multimodal loss to align fused and single-modal prediction losses.

  • Results

    95.9% accuracy on elderly activity recognition establishes ESE-FN as state of the art, improving by 2.2% over the comparison baseline.

  • Takeaways & Limitations

    ESE-FN achieves state-of-the-art performance on elderly activity recognition and remains comparable with other methods on normal action recognition.

Abstract

from arXiv · show

This work focuses on the task of elderly activity recognition, which is a challenging task due to the existence of individual actions and human-object interactions in elderly activities. Thus, we attempt to effectively aggregate the discriminative information of actions and interactions from both RGB videos and skeleton sequences by attentively fusing multi-modal features. Recently, some nonlinear multi-modal fusion approaches are proposed by utilizing nonlinear attention mechanism that is extended from Squeeze-and-Excitation Networks (SENet). Inspired by this, we propose a novel Expansion-Squeeze-Excitation Fusion Network (ESE-FN) to effectively address the problem of elderly activity recognition, which learns modal and channel-wise Expansion-Squeeze-Excitation (ESE) attentions for attentively fusing the multi-modal features in the modal and channel-wise ways. Furthermore, we design a new Multi-modal Loss (ML) to keep the consistency between the single-modal features and the fused multi-modal features by adding the penalty of difference between the minimum prediction losses on single modalities and the prediction loss on the fused modality. Finally, we conduct experiments on a largest-scale elderly activity dataset, i.e., ETRI-Activity3D (including 110,000+ videos, and 50+ categories), to demonstrate that the proposed ESE-FN achieves the best accuracy compared with the state-of-the-art methods. In addition, more extensive experimental results show that the proposed ESE-FN is also comparable to the other methods in terms of normal action recognition task.

I. INTRODUCTION

Elderly activity recognition is difficult because similar motions can differ through subtle local movements and human-object interactions. The paper proposes ESE-FN to fuse RGB and skeleton features using modal- and channel-wise attentions, with a multimodal loss for consistency.

  • Motivation: Elderly activity recognition must distinguish similar motions, subtle local movements, and human-object interactions.Examples include blowing hair versus combing hair and making a call versus playing with a mobile device.
  • Motivation: RGB and skeleton modalities must be fused to capture discriminative information from actions and interactions.
  • Proposed method: ESE-FN uses M-Net and C-Net to learn modal-wise and channel-wise Expansion-Squeeze-Excitation attentions.These networks capture modal and channel-wise dependencies among multimodal features.
  • Proposed method: The multimodal loss measures the difference between the minimum single-modal prediction loss and the fused-modality prediction loss.
  • Results: ESE-FN achieves state-of-the-art performance on elderly and normal action recognition tasks.

II. RELATED WORK

Prior action-recognition methods use RGB or skeleton data to model spatial and temporal information, but elderly activities require finer discrimination. RGB lacks temporal information, while skeleton-based methods struggle with human-object actions.

  • RGB-based Action Recognition: RGB-based methods include two-stream networks and 3D convolutional networks for modeling spatial and temporal action information.
  • RGB-based Action Recognition: RGB data alone lacks temporal information, while optical-flow computation can be too time-consuming.
  • Skeleton-based Action Recognition: Skeleton-based methods primarily learn temporal and spatial information from skeleton sequences using neural networks such as RNNs, CNNs, and LSTMs.
  • Skeleton-based Action Recognition: Skeleton-based methods have difficulty recognizing human-object actions such as blowing hair and combing hair.

C. Multi-modal Fusion

Multi-modal fusion addresses differing modality distributions and irrelevant modality-specific information by selectively modeling cross-modal dependencies. The paper revisits SENet and motivates nonlinear attention that uses expansion alongside squeeze-excitation to capture local and global information.

  • Multi-modal fusion must reinforce discriminative information while suppressing irrelevant modality-specific information across differently distributed modalities.
  • SENet captures channel-wise dependencies through nonlinear attention and has been shown to improve feature discriminability.
  • SENet first converts feature maps into a channel-wise representation through global average pooling, then uses fully connected layers to obtain channel-wise attention.
  • The attention-weighted feature map enhances discriminative information while suppressing useless or irrelevant modality-specific information to some extent.
  • Global average pooling can make SENet’s nonlinear attention sensitive to noise such as large background regions, motivating local and global spatial interaction.

B. Overview of ESE-FN

ESE-FN fuses RGB-video and skeleton features through modal-wise and channel-wise attention, then optimizes the individual and fused representations with a multi-modal loss. Its pipeline extracts, aligns, concatenates, and successively fuses the two modalities.

  • ESE-FN comprises feature extraction, modal fusion, channel fusion, and multi-modal loss components.
  • RGB videos and skeleton sequences are processed by separate backbones to obtain RGB feature fr and skeleton feature fs.
  • Two MLPs unify the sizes of fr and fs before modal-wise concatenation into a joint feature representation.
  • The joint feature is transposed, processed by M-Net and then C-Net, and summed to produce the fused multi-modal feature frs.
  • The multi-modal loss jointly uses RGB, skeleton, and fused multi-modal features to optimize all ESE-FN parameters.

C. Modal-fusion Net (M-Net)

M-Net learns modal-wise ESE attention by expanding, squeezing, and exciting concatenated multi-modal features. Its expansion captures local correlations, while pooling captures global correlations before modal-wise feature updating.

  • M-Net applies modal-wise ESE attention to aggregate local and global spatial discriminative information from multi-modal features.
  • M-Net processes its input through modal-wise expansion, squeeze, and excitation steps.
  • Stacked convolutions with different kernels expand modal information and interact local modal-wise features.
  • Average pooling converts expanded features into a global modal-wise representation used to learn M-ESEA attention.
  • Expansion and squeezing provide up-size and down-size feature interactions that capture local and global dependencies.

D. Channel-fusion Net (C-Net)

C-Net performs a second, channel-wise ESE fusion after M-Net. It uses a lighter expansion, global channel representation, and excitation to capture channel dependencies and refine the fused features.

  • C-Net learns C-ESEA to fuse multi-modal features in a channel-wise manner after M-Net processing.
  • Channel-wise expansion uses a convolutional layer to interact with and expand channel information from the local view.
  • C-Net uses only one convolution layer because cross-modal differences are smaller after M-Net, enabling a lightweight design.
  • Average pooling transforms expanded features into a global channel-wise representation.
  • Channel-wise excitation uses that representation to learn C-ESEA, update features, and capture channel-wise dependencies.

E. Multi-modal Loss (ML)

The Multi-modal Loss (ML) promotes consistency between single-modal and fused-modal predictions by comparing their recognition losses.

  • E. Multi-modal Loss (ML): ML measures the difference between RGB, skeleton, and fused-modality prediction losses to maintain consistency across representations.The losses are denoted Lr, Ls, and Lrs, respectively; each uses cross entropy.

IV. EXPERIMENTS

Experiments evaluate ESE-FN on elderly and normal action recognition, using ETRI-Activity3D and controlled training and diagnostic settings.

  • A. Datasets: ETRI-Activity3D contains 112,620 samples from 100 persons across 55 action classes, with person-ID-based training and testing splits.The dataset includes RGB videos, depth maps, and skeleton sequences collected in real-world surveillance environments.
  • B. Implementation details: Each video and skeleton sequence is converted into 64-frame inputs by sampling one frame from each of 64 clips.
  • B. Implementation details: ESE-FN training uses SGD with momentum 0.9, learning rate 0.1, weight decay 10^-4, batch size 32, and 30 epochs.
  • IV. EXPERIMENTS: Training loss and accuracy reach a steady state after about 30 epochs, while overall loss converges consistently with the modality-specific losses.

C. Diagnostic Study

Ablation studies examine the contributions of modal fusion, channel fusion, multimodal loss, and the ESE expansion design.

  • D. Ablation Study: Multi-modal baselines outperform single-modal baselines, indicating complementary information in RGB and skeleton features.
  • D. Ablation Study: Each new component improves over simple feature concatenation, and ESE-FN with all components achieves the best ablation performance.The combined M-Net and C-Net configuration improves by 0.4% over M-Net alone and 1.2% over C-Net alone.
  • D. Ablation Study: ESE-FN improves by 0.6% over SENet for modal and channel-wise fusion, supporting the effectiveness of the Expansion step.
  • D. Ablation Study: Using both modal-wise and channel-wise fusion outperforms using either fusion strategy alone for both SENet and ESE-FN variants.

E. Results and Analysis

ESE-FN achieves strong elderly activity recognition performance and improves recognition of confusing activities through discriminative multimodal information.

  • E. Results and Analysis: 95.9% accuracy makes ESE-FN state of the art on ETRI-Activity3D, improving 2.2% over FSA-CNN’s 93.7%.The reported improvement is a relative 2.3% increase over the previous state-of-the-art method.
  • E. Results and Analysis: ESE-FN is comparable to alternative methods on the NTU RGB+D normal action recognition benchmark.
  • E. Results and Analysis: ESE-FN’s confusion matrix has a lighter main diagonal than Shift-GCN and ResNeXt101, indicating better recognition of confusing elderly activities.

F. Extended Experiment on Normal Action Recognition

ESE-FN is evaluated for generalization on normal action recognition using NTU RGB+D, where it is comparable to existing methods and improves over reported state-of-the-art methods.

  • F. Extended Experiment on Normal Action Recognition: ESE-FN is evaluated on NTU RGB+D to test generalization to normal action recognition.The dataset is used as a benchmark for comparing RGB- and skeleton-based action recognition methods.
  • F. Extended Experiment on Normal Action Recognition: The NTU RGB+D evaluation uses cross-subject and cross-view data splits with separately specified training and testing samples.The supplied split description gives 16,560 testing samples for the cross-subject setting and 37,920 training versus 18,960 testing samples for the cross-view setting.
  • F. Extended Experiment on Normal Action Recognition: 0.7% and 0.2% improvements are reported over the state-of-the-art methods on NTU RGB+D.The passage reports these gains while describing ESE-FN as comparable to RNN-, CNN-, and GCN-based alternatives.
  • F. Extended Experiment on Normal Action Recognition: The comparison includes RNN-, CNN-, and GCN-based methods, including IndRNN, SK-CNN, ST-GCN, and Shift-GCN.These method families represent the alternatives used in the NTU RGB+D comparison.
  • F. Extended Experiment on Normal Action Recognition: ESE-FN combines feature extraction, modal fusion, channel-wise fusion, and multi-modal loss while learning modal and channel-wise ESE attentions.Its multi-modal loss encourages consistency between single-modal and fused multi-modal features.
Loading 2112.10992v2…