PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparative Review of Recent Kinect-based Action Recognition Algorithms

Lei Wang, Du Q. Huynh, Piotr Koniusz

arXiv:1906.09955v1cs.CV

TL;DR

Kinect action-recognition research lacked a thorough comparison of feature and representation types across evaluation settings. The paper compares ten recent algorithms on six benchmark datasets, finding stronger cross-subject than cross-view performance overall, greater cross-view robustness for skeleton features, and good deep-learning results on large datasets.

  • Problem

    Kinect-based action recognition lacked a thorough comparison of handcrafted versus deep learning and depth-based versus skeleton-based techniques.

  • Method

    The paper analyzes and compares ten Kinect-based algorithms across cross-subject and cross-view recognition using six benchmark datasets, including variants of some methods.

  • Results

    The majority of methods perform better on cross-subject than cross-view recognition, while skeleton-based features are more robust for cross-view recognition than depth-based features.

  • Takeaways & Limitations

    Handcrafted features perform better on smaller datasets, whereas deep learning methods achieve very good results when trained on large datasets.

Abstract

from arXiv · show

Video-based human action recognition is currently one of the most active research areas in computer vision. Various research studies indicate that the performance of action recognition is highly dependent on the type of features being extracted and how the actions are represented. Since the release of the Kinect camera, a large number of Kinect-based human action recognition techniques have been proposed in the literature. However, there still does not exist a thorough comparison of these Kinect-based techniques under the grouping of feature types, such as handcrafted versus deep learning features and depth-based versus skeleton-based features. In this paper, we analyze and compare ten recent Kinect-based algorithms for both cross-subject action recognition and cross-view action recognition using six benchmark datasets. In addition, we have implemented and improved some of these techniques and included their variants in the comparison. Our experiments show that the majority of methods perform better on cross-subject action recognition than cross-view action recognition, that skeleton-based features are more robust for cross-view recognition than depth-based features, and that deep learning features are suitable for large datasets.

I. INTRODUCTION

Kinect-based action recognition addresses difficult viewpoint, appearance, occlusion, and execution-variation challenges, while enabling RGB, depth, and skeleton data capture. The paper fills a comparison gap by evaluating recent methods across feature and evaluation categories.

  • Research challenges: Human action recognition remains sensitive to viewpoints, body sizes, lighting, execution speeds, and subject occlusion.These challenges affect recognition performance even when scenes are free of clutter.
  • Kinect-based recognition: Kinect captures real-time RGB and depth videos and provides toolkit-based human skeleton models for each depth-video frame.Depth data can simplify foreground segmentation in cluttered scenes because clothing colour does not affect segmentation.
  • Feature categories: Methods are grouped into handcrafted features, which require feature extraction and representation design, and deep learning features, which reduce feature engineering.Deep learning methods require substantial labelled data and long training times, and may underperform on small datasets.
  • Study contribution: The study evaluates ten recent algorithms to compare handcrafted versus deep learning and skeleton-based versus depth-based features.The authors identify a lack of thorough comparisons organized around these feature types.
  • Study contribution: The evaluation also compares cross-view and cross-subject recognition and examines camera-view effects across multiview datasets of different sizes.The study considers depth-based, skeleton-based, and combined depth+skeleton-based features.

II. RELATED WORK

Related work distinguishes depth-, skeleton-, and learning-based representations, including methods that combine modalities to capture complementary appearance, motion, and body-structure information.

  • Depth-based methods: Depth-based methods have grown with real-time, cost-effective sensors and commonly use global representations such as space-time volumes.Representative handcrafted approaches model depth geometry, motion, gradients, or spatio-temporal interest points.
  • Deep learning methods: Deep learning approaches increasingly extract high-level features from depth representations, including hierarchical depth motion maps processed by convolutional neural networks.These methods learn representations rather than relying solely on manually designed descriptors.
  • Skeleton-based methods: Skeleton-based methods include joint-based and body-part-based representations of joint positions, motion, or relative body-part geometry.Some methods express body-part relations in SE(3), while others use reference joints and frame-level distance features.
  • Deep skeleton methods: Recent skeleton methods also use CNNs, RNNs, LSTMs, and graph convolutions to model spatial structure and temporal dependencies.LSTMs are used to capture temporal dependencies and co-occurrences among human joints.
  • Multimodal methods: Combining depth and skeleton features targets object interactions, similar motion trajectories, local occlusions, and complementary modality information.Examples combine depth images, 3D joint positions, and learned body-part representations.

III. ANALYZED AND EVALUATED ALGORITHMS

The paper compares ten recent algorithms spanning handcrafted and deep learning representations derived from depth and skeleton sequences, using six benchmark datasets and multiple subject/view splits.

  • Algorithm selection: Ten algorithms from 2013 onward were selected because they use skeleton-based, depth-based, handcrafted, and/or deep learning features.The evaluated set includes HON4D, HDG, LARP-SO, SCK+DCK, HPM+TM, IndRNN, ST-GCN, and other recent methods.
  • Handcrafted algorithms: HON4D models action geometry and motion in four-dimensional spatial-coordinate, depth, and time space using quantized surface-normal distributions.The descriptor uses a 600-cell polychoron and random projector perturbations to improve classification performance.
  • Handcrafted algorithms: LARP-SO represents skeleton sequences through relative 3D body-part rotations and models actions as curves in a Lie group.A logarithm map, rolling map, and Fourier Temporal Pyramid are used to handle non-Euclidean classification, distortion, noise, and temporal misalignment.
  • Deep learning algorithms: IndRNN stacks independently recurrent neural networks, while ST-GCN constructs spatio-temporal graphs over neighboring skeleton joints.These methods represent deep learning approaches for modeling temporal and skeletal structure.
  • Implementation: The experiments combine author-provided implementations with in-house Matlab implementations and evaluate ten HDG representation variants.This setup supports comparisons across algorithms and alternative HDG feature combinations.
  • Benchmark datasets: Six benchmark datasets support cross-subject and cross-view evaluation, with NTU RGB+D used for both protocols.Cross-subject splits use half the subjects for training and testing, while UWA3D Multiview Activity II and NTU RGB+D use camera/view combinations.

B. Evaluation Settings

Evaluation settings standardize or specify feature-extraction, temporal, spatial, optimization, and input-representation choices across the benchmark experiments.

  • Feature extraction: HON4D uses 320 × 240 frames divided into 4×3×3 width-by-height-by-frame spatio-temporal cells across datasets.HOPC uses six-by-five-by-three cells with dataset-dependent spatial scales and a temporal scale of 2.
  • Feature extraction: LARP-SO uses dataset-dependent desired frame numbers for computing skeletal representations.The supplied settings identify dataset-specific temporal configuration rather than one universal frame count.
  • HDG variants: HDG variants combine histograms of depth, depth gradients, joint position differences, and joint movement volume features.The experiments evaluate ten combinations of these individual feature types.
  • Model settings: HPM+TM uses three Fourier Pyramid levels and four low-frequency coefficients, with a human pose model extracting view-invariant depth features.Average Pooling and other pooling choices were compared for this algorithm.
  • Model settings: IndRNN uses Adam with initial learning rate 2 × 10^-4, while dropout is 0.25 for cross-subject and 0.1 for cross-view experiments.The learning rate decays by 10 when evaluation accuracy does not increase.
  • Input representations: Additional IndRNN and ST-GCN experiments replace 3D skeleton inputs with joint-position-difference features centered at the torso or spine joint.This tests a translated skeleton representation as an alternative input.

C. Evaluation Measure

The paper evaluates action-recognition performance using per-class accuracy, mean accuracy across classes, and average rank across datasets.

  • Per-class accuracy is the proportion of correct labels returned for a given action class.
  • Overall dataset performance is measured by average recognition accuracy across all action classes.
  • Algorithms are ranked from 1 to 5 on each dataset according to recognition accuracy, with lower rank values indicating better performance.
  • Average Rank summarizes an algorithm’s rank values across M datasets.

D. Optimisation of Hyperparameters for HDG

The HDG experiments optimize selected hyperparameters through grid search and prune large combined feature sets using a random decision forest.

  • HDG has three hyperparameters: the number of subvolumes, the number of training trees Ntrees, and the pruning threshold θ.The subvolume count follows, while Ntrees and θ are optimized experimentally.
  • A random decision forest selects high-importance components from large combined HDG features to increase processing speed without compromising recognition accuracy.HDG-all reaches 13,250 dimensions for MSRAction3D before pruning.
  • Ntrees and θ are optimized by evaluating different combinations of individual HDG features on two datasets.MSRAction3D is used for single-view evaluation and UWA3D Multiview Activity II for cross-view evaluation.
  • The optimal Ntrees and θ values are used across datasets, whose post-pruning feature dimensions differ because their numbers of body joints differ.

V. EXPERIMENTAL RESULTS

The experiments compare handcrafted and deep-learning Kinect methods across single-view, cross-subject, and cross-view settings, showing dataset- and representation-dependent performance differences.

  • Single-view datasets: SCK+DCK outperformed all other handcrafted methods on the four single-view datasets.Its effectiveness is attributed to RBF kernels capturing higher-order statistics and action-dynamics complexity.
  • Single-view datasets: 1-layer P-LSTM with 8 video segments and 50 hidden neurons led deep-learning methods on UWA3D Activity, while 2-layer P-LSTM and HPM+TM led other datasets.The 2-layer P-LSTM led MSRAction3D, and HPM+TM led 3D Action Pairs.
  • Single-view datasets: More video segments or hidden neurons did not guarantee better performance because noisy frames and overfitting can weaken feature representation and training.
  • Single-view datasets: Using jpd instead of raw 3D joint coordinates improved IndRNN and ST-GCN on almost all datasets, including gains of 4.56% and 32.97% for specified IndRNN settings.The gains were observed for 6-layer IndRNN on MSRAction3D and 4-layer IndRNN on CAD-60, respectively.
  • Single-view datasets: SCK+DCK achieved average rank 1 across the four single-view datasets, followed by 6-layer IndRNN with jpd at average rank 2.ST-GCN averaged rank 4.75, consistent with its more complex architecture requiring larger training data.
  • NTU RGB+D: On NTU RGB+D, SCK+DCK was the best handcrafted method for both cross-subject and cross-view recognition, while combining HDG features improved accuracy.
  • NTU RGB+D: With jpd, ST-GCN reached 83.36% for cross-subject recognition and 6-layer IndRNN reached 89.0% for cross-view recognition.For ST-GCN in cross-subject recognition, jpd increased top-1 and top-5 accuracy by 1.79% and 0.61%.
  • NTU RGB+D: On NTU RGB+D, deeper RNN, LSTM, and P-LSTM variants outperformed one-layer variants, whereas HPM methods underperformed because their pose dictionary was less complex than the dataset’s actions.

C. UWA3D Multiview Activity II Dataset

On UWA3D Multiview Activity II, cross-view performance varies with feature type, model complexity, and training-data availability.

  • Cross-view recognition: HDG-all features performed best among handcrafted methods for cross-view recognition.
  • Cross-view recognition: Skeleton-only methods generally outperformed depth-based methods in the reported ranking.HDG-jpd+jmv ranked second among skeleton-only methods, followed by SCK+DCK, HDG-jmv, and LARP-SO-FTP; these exceeded HON4D and HDG depth variants.
  • Cross-view recognition: Adding skeleton-based jpd and/or jmv features to HDG improved its cross-view results.
  • Deep-learning methods: HPM+TM and HPM+AP achieved the highest results among deep-learning methods, while Clips+CNN+MTLN was limited by the dataset’s small number of video samples.
  • Deep-learning methods: Stacking more IndRNN layers or using jpd for IndRNN and ST-GCN did not improve accuracy, which the authors attribute to insufficient representative training videos.
  • Error analysis: The HDG-all confusion matrix shows errors among actions with similar motion trajectories or appearances.Examples include one-hand versus two-hand waving, walking versus irregular walking, and bending versus putting down.

VI. DISCUSSIONS

Across the evaluated algorithms, performance varied between single-view, cross-subject, and cross-view recognition, with cross-view results generally weaker. SCK+DCK led cross-subject recognition, while HDG-all, SCK+DCK, and improved IndRNN with jpd features shared strong cross-view performance.

  • The HDG-all confusion matrix shows errors between actions with similar motion trajectories or visual appearances.Examples include one- versus two-hand waving, walking versus irregular walking, and bending versus putting down.
  • SCK+DCK outperformed all other algorithms for cross-subject action recognition.
  • HON4D, LARP-SO-FTP, Clips+CNN+MTLN, P-LSTM, and SCK+DCK performed better for single-view than cross-view recognition.
  • Average cross-view recognition accuracy was lower than cross-subject accuracy for most feature categories, except deep learning depth-based methods.The two methods in that exception performed well on UWA3D Multiview Activity II and reasonably well on NTU RGB+D.

B. Influence of camera views in cross-view evaluation

Camera-view combinations substantially influenced cross-view recognition, with performance depending on the geometric relationship between training and testing views and on feature type. Recognition was high when V2 and V3 trained the model and V1 was tested, while several depth-based handcrafted combinations performed poorly.

  • Influence of camera views in cross-view evaluation: The comparison groups algorithms by depth-based, skeleton-based, and depth+skeleton-based representations, while distinguishing handcrafted from deep learning features.
  • Influence of camera views in cross-view evaluation: Recognition accuracy was high when V2 and V3 were used for training and V1 was used for testing.The front view V1 lies between the left view V2 and right view V3.
  • Influence of camera views in cross-view evaluation: V2V4-V1 and V3V4-V1 produced slightly higher recognition accuracy than V2V3-V1.
  • Influence of camera views in cross-view evaluation: For handcrafted depth-based methods, accuracy dropped most when V2 and V4 trained the model and V3 was tested.The authors attribute this to different visual appearances between V2 and V3 and difficulty finding view-invariant features.
  • Influence of camera views in cross-view evaluation: Other view combinations, including V1V3-V4, V1V3-V2, and V1V2-V4, also led to lower performance for handcrafted depth-based methods.

C. Depth-based features versus skeleton-based features

Skeleton-based features generally outperformed or were more robust than depth-based features across cross-subject and cross-view evaluations. Adding depth to skeleton features could reduce cross-subject accuracy, and no handcrafted-versus-deep-learning comparison was available for depth+skeleton features.

  • Depth-based features versus skeleton-based features: Skeleton-based features outperformed depth-based features for cross-subject recognition in both handcrafted and deep learning categories.
  • Depth-based features versus skeleton-based features: Adding depth-based features to skeleton-based features slightly decreased cross-subject recognition accuracy.The authors associate this decrease with background clutter and noise in depth sequences.
  • Depth-based features versus skeleton-based features: Skeleton-based algorithms using high-confidence human-joint features performed well in human action recognition.LARP-SO-FTP and its variants are cited as examples.
  • Depth-based features versus skeleton-based features: Skeleton-based features produced better cross-view results than depth-based features, and HDG pruning supported their greater robustness.
  • Depth-based features versus skeleton-based features: No handcrafted-versus-deep-learning comparison was possible for depth+skeleton features because the experiments included no deep learning methods in that category.
  • Depth-based features versus skeleton-based features: Deep learning methods performed better on the large NTU RGB+D dataset but lower on the other smaller datasets.NTU RGB+D contains more than 56,000 video sequences.
  • Depth-based features versus skeleton-based features: Most handcrafted methods were highly dataset-dependent and features designed for one dataset were not necessarily transferable to another.

E. ‘Quo Vadis, action recognition?’

The paper reviews evolving Kinect-based action-recognition representations and compares ten algorithms across six benchmark datasets. Its results favor skeleton-based robustness, handcrafted features on smaller datasets, and deep learning when large training datasets are available, while emphasizing that more robust methods remain needed for new environments.

  • E. ‘Quo Vadis, action recognition?’: Handcrafted representations progressed from global descriptors to local descriptors and then combined global-local representations.Examples include HON4D, HOPC, and HDG.
  • E. ‘Quo Vadis, action recognition?’: Recent methods increasingly design robust 3D human joint-based representations and use skeleton sequences because they are easier to process and analyze than depth videos.
  • E. ‘Quo Vadis, action recognition?’: The study compares ten state-of-the-art algorithms using handcrafted and deep learning features computed from depth and skeleton sequences across six benchmark datasets.
  • E. ‘Quo Vadis, action recognition?’: Skeleton-based features were more robust than depth-based features for both cross-subject and cross-view recognition.
  • E. ‘Quo Vadis, action recognition?’: Handcrafted features performed better on smaller datasets, whereas deep learning methods achieved very good results on large datasets.
  • E. ‘Quo Vadis, action recognition?’: The authors conclude that new and more robust action-recognition algorithms remain required for real and new environments.
Loading 1906.09955v1…