PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Action Recognition from Various Data Modalities: A Review

Zehua Sun, Qiuhong Ke, Hossein Rahmani, Mohammed Bennamoun, Gang Wang, Jun Liu

arXiv:2012.11866v5cs.CV

TL;DR

HAR must recognize actions across modalities whose information and application-specific advantages differ. This survey reviews deep-learning methods for single and multiple modalities, including fusion and co-learning, and also covers benchmarks and future directions. It highlights the need for more effective multimodal strategies and broader datasets because existing methods face challenges such as over-fitting and controlled data collection.

  • Problem

    HAR uses heterogeneous modalities with distinct strengths, motivating a unified review of single-modality methods and multimodal approaches.

  • Method

    The paper surveys deep-learning HAR methods by modality, covering single-modal architectures, multimodal fusion and co-learning, benchmark datasets, and future research directions.

  • Results

    The survey synthesizes methods across numerous modalities and reports benchmark datasets while discussing potential research directions.

  • Takeaways & Limitations

    Multimodal fusion can improve HAR performance, co-learning can address missing modality data, and further work is needed on effective strategies and practical benchmarks.

  • Takeaways & Limitations

    Event streams remain difficult to use directly because their asynchronous data cannot generally be processed effectively by conventional video-analysis techniques.

Abstract

from arXiv · show

Human Action Recognition (HAR) aims to understand human behavior and assign a label to each action. It has a wide range of applications, and therefore has been attracting increasing attention in the field of computer vision. Human actions can be represented using various data modalities, such as RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi signal, which encode different sources of useful yet distinct information and have various advantages depending on the application scenarios. Consequently, lots of existing works have attempted to investigate different types of approaches for HAR using various modalities. In this paper, we present a comprehensive survey of recent progress in deep learning methods for HAR based on the type of input data modality. Specifically, we review the current mainstream deep learning methods for single data modalities and multiple data modalities, including the fusion-based and the co-learning-based frameworks. We also present comparative results on several benchmark datasets for HAR, together with insightful observations and inspiring future research directions.

1 INTRODUCTION

HAR supports applications ranging from surveillance and autonomous navigation to retrieval, human-robot interaction, and entertainment. This review organizes recent deep-learning HAR methods by data modality, covering single modalities, multimodal fusion and co-learning, datasets, and future directions.

  • HAR recognizes and understands human actions for surveillance, autonomous navigation, video retrieval, human-robot interaction, and entertainment.
  • Recent HAR research uses RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi modalities.Different modalities offer distinct advantages depending on application scenarios.
  • Multimodal approaches are categorized into fusion-based methods that combine modalities and co-learning-based methods that transfer knowledge across modalities.Fusion combines information from two or more modalities, whereas co-learning uses one modality to strengthen another.
  • The review covers mainstream deep-learning architectures for single modalities and methods using multiple modalities for enhanced HAR.
  • The survey focuses mainly on short, trimmed video segments containing one action instance and reports benchmark datasets.

2 SINGLE MODALITY

Single-modality HAR methods use modality-specific deep-learning architectures, with RGB research organized around two-stream CNNs, RNNs, 3D CNNs, and Transformers. RGB methods exploit appearance and motion but face challenges from visual variation, computational cost, and long-term temporal modeling.

  • Single-modality HAR research covers RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi inputs.
  • RGB modality: RGB recognition is challenged by background, viewpoint, scale, and illumination variations, large data sizes, high computational cost, and limited long-term dependency modeling in two-stream architectures.
  • RGB modality: RGB video methods are grouped into two-stream 2D CNN, RNN, 3D CNN, and Transformer-based architectures.
  • RGB modality: Two-stream CNNs process separate appearance and motion inputs through distinct networks before fusing their outputs for classification.Classic variants use RGB frames and optical flow, while extensions address long-term information, motion saliency, and optical-flow computation.
  • RGB modality: RNN-based RGB methods commonly extract frame-level features with 2D CNNs and model temporal dynamics using LSTMs.Gated RNNs are used because vanilla RNNs struggle with long-term dependencies.

2.2 SKELETON MODALITY

Skeleton data represents human actions through body-joint trajectories and offers a simple, efficient, and informative pose representation. Deep-learning methods model skeleton sequences with RNNs, CNNs, graph networks, and Transformers, but sparse, noisy data and missing shape information remain challenges.

  • Skeleton sequences encode human body-joint trajectories and can be obtained from RGB or depth videos, or motion-capture systems.
  • Skeleton data provides body structure and pose information, with scale invariance and robustness to clothing textures and backgrounds.
  • Deep-learning skeleton HAR methods are organized into RNN, CNN, GNN, and Transformer categories.
  • RNN and CNN methods: RNNs and LSTMs model temporal context, while CNN methods transform skeleton sequences into pseudo-images or apply temporal convolution.Pseudo-image methods encode spatio-temporal information into color-texture images for CNN processing.
  • GNN or GCN methods: Graph-based methods represent joint dependencies as graphs because vector sequences and image maps may not fully model complex spatio-temporal joint correlations.GCN methods can learn spatial and temporal patterns from skeleton data.
  • Skeleton HAR remains challenged by sparse representations, noisy skeleton information, and missing shape information important for human-object interactions.Depth maps are discussed as a way to provide 3D geometric structure while retaining shape information.

2.3 DEPTH MODALITY

Depth maps encode viewpoint-based distance information and provide geometric structure useful for human action recognition. The surveyed methods mainly use active-sensor depth videos, while fusion with other modalities addresses missing appearance information.

  • Depth representation: Depth maps represent distance from a viewpoint and provide reliable 3D structural and geometric shape information for human action recognition.They are produced using active sensors such as time-of-flight and structured-light cameras or passive stereo cameras.
  • Depth acquisition: Most reviewed depth-based methods use videos captured by low-cost, reliable active sensors, while only a few use stereo-camera depth maps.The survey therefore focuses on active-sensor depth videos.
  • Deep learning methods: Deep learning models have outperformed hand-crafted feature-based methods for human action recognition from depth maps.Representative approaches use weighted hierarchical depth motion maps, structured dynamic images, and multiple dynamic depth representations.
  • Modeling strategies: Depth-based approaches address view variation through view-invariant pose modeling and temporal pyramids, while other work estimates depth from RGB videos before classification.These methods extend depth-based recognition beyond directly captured depth data.
  • Multimodal use: Depth is often fused with other modalities because it provides geometric information but lacks appearance information useful in some scenarios.The survey places such multimodal methods in its fusion discussion.

2.4 INFRARED MODALITY

Infrared sensing supports human action recognition without relying on external ambient light, making it especially suitable for nighttime settings. Deep learning methods model spatial and temporal information, but low contrast and signal-to-noise ratio can limit robustness.

  • Infrared sensing: Infrared sensors do not require external ambient light, making infrared data particularly suitable for nighttime human action recognition.Active infrared sensors emit rays, whereas passive thermal sensors detect heat energy emitted by targets.
  • Deep learning methods: Deep learning methods process infrared sequences with CNN-LSTM and 3D CNN architectures to model spatio-temporal information.One reported 3D CNN approach achieved real-time human action recognition.
  • Multi-stream methods: Two-stream architectures combine infrared appearance data with optical-flow information to capture complementary appearance and motion cues.The reviewed designs include two-stream 3D CNN frameworks.
  • Limitations: Infrared images may have low contrast and low signal-to-noise ratio, challenging robust human action recognition in some scenarios.

2.5 POINT CLOUD MODALITY

Point clouds represent subjects’ spatial distribution, surface characteristics, silhouettes, and 3D geometry for human action recognition. Deep learning methods have advanced from voxelized representations toward direct point processing and transformer-based architectures, while noise and non-uniform sampling remain challenges.

  • Point-cloud representation: Point clouds capture spatial distribution, surface characteristics, silhouettes, and 3D geometric shapes that support human action recognition.They can be obtained with 3D sensors such as LiDAR and Kinect or through image-based 3D reconstruction.
  • Deep learning methods: Deep learning is the mainstream approach for point-cloud action recognition and generally achieves better performance than early hand-crafted descriptor methods.
  • Point processing: Voxelizing point-cloud sequences enables 3D action encoding but introduces quantization errors and inefficient processing.MeteorNet instead directly stacks multi-frame point clouds without voxel quantization.
  • Transformer methods: Recent point-cloud action recognition methods use transformers with attention modules and representative-point sampling.The Point Attention Transformer uses Group Shuffle Attention and Gumbel Subset Sampling.
  • Advantages and limitations: Point clouds can reduce sensitivity to viewpoint variation, but noise, non-uniform point distributions, and processing all points challenge robust recognition.Viewpoint normalization can be performed by rotating point clouds in 3D space.

2.6 EVENT STREAM MODALITY

Event cameras asynchronously record per-pixel illumination changes at microsecond temporal scales, offering low-latency, low-power action sensing with reduced background redundancy. Their sparse asynchronous output remains difficult to process directly with conventional video techniques.

  • Event representation: Event cameras respond to illumination changes independently at each pixel rather than capturing complete image arrays.They generate asynchronous events and can operate at a microsecond temporal scale.
  • Advantages: Event cameras offer high dynamic range, low latency, low power consumption, and reduced motion blur for human action recognition.They can filter background information while retaining foreground movement.
  • Event aggregation: Many methods convert asynchronous event streams into synchronous frames before applying conventional computer vision or CNN-LSTM techniques.Examples include binary representations accumulated over fixed time intervals and timestamp-image encodings.
  • Direct processing: Other deep learning approaches directly process event data, including unsupervised spatio-temporal filtering followed by 3D CNNs and spiking neural networks.
  • Limitations: Directly using asynchronous event streams remains challenging because event data is computationally cheap but sparse, asynchronous, and poorly suited to conventional video analysis.

2.7 AUDIO MODALITY

Audio is useful for locating actions in synchronized video streams and can complement visual modalities, but audio-only HAR remains uncommon because the signal lacks sufficient information for accurate recognition.

  • Audio-video synchronization enables temporal action localization, reducing human labeling effort and computational costs.
  • Only a few deep learning methods have addressed HAR from audio alone.
  • Audio-only HAR is uncommon because audio signals do not contain enough information for accurate recognition.
  • Multimodal methods commonly use audio as complementary information for more reliable and efficient HAR.

2.8 ACCELERATION MODALITY

Acceleration signals support robust and fine-grained HAR, including privacy-preserving monitoring, but wearable sensors can be cumbersome and their placement affects performance.

  • Accelerometers provide robustness against occlusion, viewpoint, lighting, and background variations for HAR.Tri-axial accelerometers estimate acceleration along the x, y, and z axes.
  • Deep learning methods process tri-axial acceleration with CNNs, CNN–Bi-LSTM networks, or transformed recurrence-plot representations.
  • Acceleration is suited to fine-grained HAR and action monitoring, especially for elderly care because it protects subject privacy.
  • Wearable acceleration sensors may be cumbersome or disturbing, and sensor placement can affect HAR performance.

2.9 RADAR MODALITY

Radar-based HAR exploits motion and structural information while offering illumination robustness, privacy protection, and through-wall sensing, but radar cost and real-world complexity remain barriers.

  • Radar spectrograms encode target motion and structure through Doppler signatures, while FMCW radar also measures target distances.
  • Radar supports HAR despite illumination and weather variation, protects privacy, and can operate through walls.
  • Deep learning approaches use micro-Doppler spectrograms or raw radar range data with CNNs, auto-correlation, and Random Forest classifiers.
  • Two-stream CNNs combine micro-Doppler with range spectrograms or radar echoes to represent complementary target signatures.
  • RNN-based methods interpret micro-Doppler spectrograms as temporal sequences using LSTM or stacked RNN architectures.
  • Radar systems are relatively expensive, and future work must address more complex actions in real-world scenarios.

2.10 WIFI MODALITY

WiFi-based HAR uses reflected signal changes, especially CSI, to provide convenient, low-cost, privacy-preserving sensing, but phase–amplitude use and dynamic-environment robustness remain challenging.

  • Human bodies reflect WiFi signals, enabling HAR and sometimes through-wall HAR with convenient, simple, privacy-preserving, low-cost devices.
  • Most WiFi-based HAR methods use Channel State Information, whose reflected-signal variations encode action-related changes.CSI is fine-grained information computed from raw WiFi signals at the receiver.
  • Deep learning methods model CSI using sparse auto-encoders, temporal convolutional architectures, CNN–Bi-LSTM pipelines, and attention-based Bi-LSTMs.
  • WiFi HAR also transforms CSI into radio images or classifies each distortion sample for fine-grained action recognition.
  • Open challenges include using CSI phase and amplitude more effectively and improving robustness in dynamic environments.

3 MULTI-MODALITY

Multi-modal HAR combines complementary sensory information through fusion or transfers knowledge across modalities through co-learning. The survey reviews these frameworks, their applications across visual and non-visual modalities, and the continuing challenge of computationally efficient fusion.

  • 3 MULTI-MODALITY: Multi-modal learning aggregates information from multiple modalities through fusion or transfers knowledge between modalities through co-learning.Fusion is used during training and inference, whereas co-learning uses auxiliary modalities during training and can support cases where modalities are missing at testing.
  • 3.1 FUSION: Score fusion combines modality-specific decisions, while feature fusion combines learned representations before action classification.Score fusion may use weighted averaging or a learned score-fusion model.
  • 3.1 FUSION: Visual fusion methods combine RGB, depth, skeleton, infrared, optical flow, or point-cloud representations using CNNs, RNNs, LSTMs, and other architectures.Examples include four-stream RGB-depth networks, skeleton-RGB RNN-CNN systems, and networks fusing pose, motion, and raw RGB streams.
  • 3.1 FUSION: Audio-visual fusion uses audio as complementary information to visual appearance and motion for human action recognition.The reviewed methods include multi-stream CNNs, semantic label embeddings, modality dropout, audio-adaptive representations, and multimodal video Transformers.
  • 3.3 DISCUSSION: Effective modality fusion remains open because many existing multi-modality methods use complicated architectures with high computational costs.Efficient multi-modality HAR therefore remains an important research need.
  • 3.2 CO-LEARNING: Co-learning transfers knowledge from auxiliary modalities to strengthen another modality, with auxiliary data required during training rather than testing.Reviewed examples transfer knowledge between RGB and depth, RGB and skeleton, or non-visual signals and RGB through supervision or distillation.

4 DATASETS

The survey catalogs benchmark datasets used for HAR across RGB, 3D, and multi-modality settings. These datasets provide evaluation resources spanning diverse sensor combinations and action-recognition tasks.

  • 4 DATASETS: Benchmark datasets are compiled with attributes covering multiple data modalities for training and evaluating HAR methods.The survey presents a representative dataset list and summarizes dataset attributes.
  • 4 DATASETS: RGB-based HAR commonly uses UCF101, HMDB51, Kinetics-400, Kinetics-600, Kinetics-700, EPIC-KITCHENS-55, THUMOS Challenge 15, ActivityNet, and Something-Something-v1.These datasets support evaluation of RGB-based action-recognition methods.
  • 4 DATASETS: NTU RGB+D and NTU RGB+D 120 are widely used large-scale benchmarks for 3D skeleton, depth, infrared, and point-cloud HAR.The survey also identifies additional datasets for these modalities, including MSRDailyActivity3D.
  • 4 DATASETS: Multi-modality HAR commonly uses NTU RGB+D, NTU RGB+D 120, MMAct, and EPIC-KITCHENS as large benchmark datasets.MSRDailyActivity3D, UTD-MHAD, and PKU-MMD are also popularly used for multi-modality research.

5 DISCUSSION

The discussion identifies dataset quality, multimodal learning, efficiency, early recognition, few-shot learning, and label-efficient learning as important directions for HAR. It emphasizes practical constraints including limited data, over-fitting, and computational cost.

  • Datasets: New large and comprehensive HAR benchmarks remain necessary because dataset size, diversity, applicability, and modality type affect research development.The survey also highlights crowded-environment, group-action, and human-human-interaction datasets as further needs.
  • Multi-modality Learning: Existing multimodality methods can be less effective than expected because of challenges such as over-fitting.The survey calls for more effective fusion and co-learning strategies.
  • Efficient Action Analysis: Efficient HAR remains important because strong performance often relies on high computational complexity and resource consumption.Future work should reduce CPU, GPU, and energy costs while achieving efficient and fast action recognition.
  • Early Action Recognition: Early action recognition targets recognizing an action before it is fully performed for applications such as online human-robot interaction and early alarms.The task is also called action prediction.
  • Few-shot Action Analysis: Few-shot action analysis addresses the difficulty of collecting large training sets, especially for multi-modality action classes.The survey identifies more advanced few-shot HAR as an open direction for practical data-scarce scenarios.
  • Unsupervised and Semi-supervised Learning: Unsupervised and semi-supervised learning can reduce reliance on expensive labels by using unlabelled action data for model training.These approaches are motivated by the greater ease of collecting unlabelled samples than labeled data.

6 CONCLUSION

The paper provides a comprehensive review of HAR methods across data modalities, including single-modality and multi-modality recognition. It also surveys benchmark datasets and discusses potential future research directions.

  • 6 CONCLUSION: The survey reviews HAR methods using different data modalities and covers multi-modality recognition through fusion and co-learning.Its scope encompasses the major modality-based method categories discussed throughout the paper.
  • 6 CONCLUSION: Benchmark datasets are reviewed alongside potential research directions for human action recognition.The conclusion summarizes the survey’s dataset coverage and forward-looking discussion.
Loading 2012.11866v5…