PaperlayerView exact revision on arXiv

Source-linked AI summary

ACTION-Net: Multipath Excitation for Action Recognition

Zhengwei Wang, Qi She, Aljosa Smolic

arXiv:2103.07372v1cs.CV

TL;DR

Video understanding needs high accuracy with low computation cost. ACTION-Net addresses this by adding a lightweight multipath ACTION module to 2D CNNs, with consistently improved performance over 2D CNN counterparts across three backbones and datasets.

  • Problem

    Video understanding faces the challenge of performing action recognition at high accuracy and low computation cost.

  • Method

    ACTION-Net equips 2D CNNs with a plug-and-play ACTION module that uses multipath excitation to process spatio-temporal, channel-wise, and motion features inside one network.

  • Results

    ACTION-Net consistently improves over 2D CNN counterparts on ResNet-50, BNInception, and MobileNet V2 across Something-Something V2, Jester, and EgoGesture.

  • Takeaways & Limitations

    The module provides a generic way to enhance 2D CNN video action recognition while introducing limited extra computation.

Abstract

from arXiv · show

Spatial-temporal, channel-wise, and motion patterns are three complementary and crucial types of information for video action recognition. Conventional 2D CNNs are computationally cheap but cannot catch temporal relationships; 3D CNNs can achieve good performance but are computationally intensive. In this work, we tackle this dilemma by designing a generic and effective module that can be embedded into 2D CNNs. To this end, we propose a spAtio-temporal, Channel and moTion excitatION (ACTION) module consisting of three paths: Spatio-Temporal Excitation (STE) path, Channel Excitation (CE) path, and Motion Excitation (ME) path. The STE path employs one channel 3D convolution to characterize spatio-temporal representation. The CE path adaptively recalibrates channel-wise feature responses by explicitly modeling interdependencies between channels in terms of the temporal aspect. The ME path calculates feature-level temporal differences, which is then utilized to excite motion-sensitive channels. We equip 2D CNNs with the proposed ACTION module to form a simple yet effective ACTION-Net with very limited extra computational cost. ACTION-Net is demonstrated by consistently outperforming 2D CNN counterparts on three backbones (i.e., ResNet-50, MobileNet V2 and BNInception) employing three datasets (i.e., Something-Something V2, Jester, and EgoGesture). Codes are available at \url{https://github.com/V-Sense/ACTION-Net}.

1. Introduction

Video action recognition must capture spatial, temporal, channel-wise, and motion information while balancing accuracy with computational cost. ACTION-Net addresses this balance by embedding a lightweight multipath ACTION module into 2D CNNs.

  • Video understanding supports applications including VR/AR and video-sharing services but faces demands for high accuracy and low computation cost.
  • 2D CNNs provide lightweight inference but lack temporal modeling, while TSM adds temporal shifts yet lacks explicit action-motion modeling.
  • The module is plug-and-play and supports 2D CNN backbones including ResNet-50, BNInception, and MobileNet V2.
  • ACTION combines spatio-temporal, channel-wise, and motion excitation paths to process multiple information types within a single network at feature level.
  • ACTION-Net demonstrates superior performance on Something-Something V2, Jester, and EgoGesture compared with 2D CNN counterparts.

2. Related Works

Prior work spans computationally heavy 3D CNNs and lightweight 2D CNNs, with embedded modules extending 2D models toward temporal and motion modeling. Channel-excitation methods also require adaptation because image-oriented designs omit video temporal properties.

  • 2.1. 3D CNN-based Framework: 3D CNNs model spatio-temporal information effectively but introduce massive parameters, overfitting and convergence concerns, and heavier computation than 2D CNNs.
  • 2.2. 2D CNN-based Framework: TSN applies 2D CNNs to sparsely sampled video segments, but direct 2D processing lacks temporal modeling for video sequences.
  • 2.2. 2D CNN-based Framework: TSM adds temporal information by shifting part of the channels, yet it lacks explicit modeling of differences among neighboring frames.
  • 2.2. 2D CNN-based Framework: Embedded modules such as MFNet, TEINet, TEA, STM, and GSM extend 2D CNNs with motion, temporal, or spatiotemporal modeling.
  • 2.3. Channel-wise Features: SENet enhances channel-wise image features through channel interdependencies, but processes images independently without considering video temporal properties.

3. Design of ACTION

ACTION combines spatio-temporal, channel-wise, and motion excitation paths to enrich 2D CNN features while preserving the original network components. STE uses efficient channel-averaged 3D convolution, CE models temporal channel interdependencies, and ME derives motion from adjacent-frame feature differences.

  • Spatio-Temporal Excitation (STE): STE averages features across channels, applies a 3 × 3 × 3 3D convolution to the reshaped tensor, and produces a spatio-temporal mask for channel-wise excitation.The mask is applied element-wise across all input channels.
  • Spatio-Temporal Excitation (STE): STE is more computationally efficient than conventional 3D convolution because its 3D convolution processes channel-averaged features.Each input channel receives the refined spatio-temporal excitation.
  • Channel Excitation (CE): CE performs spatial average pooling, reduces channels by a ratio of r = 16, and inserts a kernel-size-3 1D convolution between channel transformations for temporal reasoning.The resulting channel mask recalibrates the input features through element-wise excitation.
  • ACTION-Net integration: ACTION-Net inserts the ACTION module at the beginning of each residual block without modifying the original block components.The architecture samples one frame randomly from each of T equally divided video segments.
  • Motion Excitation (ME): ME models motion from adjacent-frame feature differences, concatenates the resulting temporal features with zero padding for the final element, and generates a channel mask.It uses the same squeeze, spatial pooling, and mask-generation strategy as CE.
  • ACTION module overview: ACTION consists of Spatio-Temporal Excitation, Channel Excitation, and Motion Excitation sub-modules whose outputs are combined by element-wise addition.The resulting feature can perceive spatio-temporal information, channel interdependencies, and motion.

4. Experiments

Experiments evaluate ACTION-Net across three datasets, compare it with 2D CNN baselines and state-of-the-art methods, and analyze excitation paths, module placement, and computational efficiency.

  • Improving Performance of 2D CNNs: ACTION-Net consistently outperforms TSN and TSM across EgoGesture, Something-Something V2, and Jester.The fair comparison uses ResNet-50 with 8 input frames.
  • Improving Performance of 2D CNNs: 2%, 3.8%, and 2.7% Top-1 accuracy improvements over TSM occur on EgoGesture, Something-Something V2, and Jester, respectively.
  • Comparisons with State-of-the-Art: ACTION-Net achieves state-of-the-art performance on Jester and EgoGesture, including with 8 RGB frames as input.
  • Ablation Study: Each excitation path improves 2D CNN baselines with limited added computation, while ME adds more computation and parameters than STE and CE.STE and CE add negligible extra computation; ME captures temporal differences among adjacent frames.
  • Ablation Study: STE, CE, ME, and ACTION-Net have efficiency η values of 0.18%, 0.29%, 2.83%, and 2.52%, respectively, with STE the most efficient.Smaller η indicates fewer extra FLOPs introduced per 1% Top-1 accuracy increase.

5. Conclusion

ACTION-Net is a plug-and-play module for 2D CNNs that uses multipath excitation to model spatio-temporal, channel-wise, and motion features. Across three backbones and datasets, it improves on 2D CNN counterparts with limited extra computation.

  • ACTION-Net inserts multipath excitation into 2D CNNs to process spatio-temporal, channel-wise, and motion features.
  • The architecture is demonstrated with ResNet-50, BNInception, and MobileNet V2 on three large-scale action-recognition datasets.
  • ACTION-Net consistently improves over 2D CNN counterparts while introducing limited extra computation.

A. Backbone Architecture in Experiments

The experiments embed ACTION or TSM at the start of backbone blocks, using backbone-specific placement patterns across ResNet-50, BNInception, and MobileNet V2.

  • The study keeps the number and inserted position of TSM and ACTION consistent across the three backbones.
  • ResNet-50: ResNet-50 receives ACTION or TSM at the start of every residual block from res2 through res5, for 16 modules total.
  • BNInception: BNInception receives ACTION or TSM at the starting point of each inception block, adding 10 modules.
  • MobileNet V2: MobileNet V2 receives ACTION or TSM at the start of each selected bottleneck, including two stage4 blocks and the first stage5 block.

B. The Location of ACTION

The location ablation compares possible insertion points for ACTION across three backbones and identifies the beginning of each block as the most effective default.

  • Inserting ACTION at the beginning of each block, designated Loc 1, is more effective than other tested locations for all three backbones.
  • The ablation tests four possible locations for ResNet-50 and MobileNet V2, but only two for BNInception.
  • Table 1 reports Top-1 accuracy on EgoGesture using 8 segments, with Loc 1 as the default main-paper configuration.

C. Visualization Results

The visualizations compare TSN, TSM, ACTION-Net, and its three excitation sub-modules across three hand-action examples. ACTION-based methods produce smoother action-movement CAMs than the baselines.

  • The visualizations compare TSN, TSM, STE, CE, ME, and ACTION-Net on ‘Rotate fists counterclockwise’, ‘Applaud’, and ‘Draw circle with hand in horizontal surface’.
  • TSN and TSM mainly recognize objects but do not produce smooth CAMs for movement across the presented video sequences.
  • ACTION-Net and each excitation sub-module extract temporal information by producing smoother CAMs for the action movement than TSN and TSM.
  • ME produces the smoothest CAM for action movement between adjacent frames, while STE and CE show more limited temporal modeling.
Loading 2103.07372v1…