PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Pooling Based Large-scale 3D Action Recognition with Convolutional Neural Networks

Pichao Wang, Wanqing Li, Zhimin Gao, Chang Tang, Philip Ogunbona

arXiv:1804.01194v2cs.CV

TL;DR

Depth action recognition needs compact representations that capture spatial-temporal and 3D structural information without requiring large-scale training from scratch. The paper constructs three dynamic depth-image representations with hierarchical bidirectional rank pooling and uses them with ConvNets, achieving state-of-the-art results across three large datasets using depth alone.

  • Problem

    Effective video representations for feeding depth sequences into deep neural networks remain unclear, especially when exploiting both temporal dynamics and depth-based structure.

  • Method

    The paper constructs DDI, DDNI, and DDMNI from segmented depth sequences using hierarchical bidirectional rank pooling, then applies ConvNets for action recognition.

  • Results

    The proposed representations achieved state-of-the-art results on ChaLearn LAP continuous and isolated gesture recognition and NTU RGB+D using depth modality only.

  • Takeaways & Limitations

    The representations provide compact depth-sequence inputs that support ConvNet fine-tuning while capturing posture dynamics and 3D structural information.

Abstract

from arXiv · show

This paper proposes three simple, compact yet effective representations of depth sequences, referred to respectively as Dynamic Depth Images (DDI), Dynamic Depth Normal Images (DDNI) and Dynamic Depth Motion Normal Images (DDMNI), for both isolated and continuous action recognition. These dynamic images are constructed from a segmented sequence of depth maps using hierarchical bidirectional rank pooling to effectively capture the spatial-temporal information. Specifically, DDI exploits the dynamics of postures over time and DDNI and DDMNI exploit the 3D structural information captured by depth maps. Upon the proposed representations, a ConvNet based method is developed for action recognition. The image-based representations enable us to fine-tune the existing Convolutional Neural Network (ConvNet) models trained on image data without training a large number of parameters from scratch. The proposed method achieved the state-of-art results on three large datasets, namely, the Large-scale Continuous Gesture Recognition Dataset (means Jaccard index 0.4109), the Large-scale Isolated Gesture Recognition Dataset (59.21%), and the NTU RGB+D Dataset (87.08% cross-subject and 84.22% cross-view) even though only the depth modality was used.

I. INTRODUCTION

Depth-based action recognition benefits from geometric cues and illumination robustness, but effectively representing video for deep classification remains unresolved. The paper extends rank pooling to depth sequences and introduces hierarchical, bidirectional representations for broader action-recognition settings.

  • Video representation for deep action recognition remains unclear, with alternatives including frame sequences, 3D ConvNets, encoders, and recurrent networks.
  • The method extends rank pooling to depth sequences because direct pooling does not fully exploit their spatial-temporal and structural information.
  • Hierarchical bidirectional rank pooling captures higher-order and nonlinear depth dynamics while reducing conventional rank pooling’s bias toward past frames.
  • The paper extends the earlier work to continuous recognition, evaluates NTU RGB+D alongside ChaLearn datasets, and reports state-of-the-art results using depth alone.

B. Depth Based Action Recognition

Depth-based recognition methods encode spatial, temporal, or structural information using hand-crafted features, deep networks, or structured images. Existing approaches can require substantial training data or skeleton information, motivating image-based depth representations compatible with ConvNets.

  • Existing depth-recognition methods use representations such as 3D point sets, depth motion maps, oriented normals, and structured images to encode action information.
  • Some depth-recognition approaches are limited by noise in fine-grained actions, large training-data requirements, or dependence on skeleton data.
  • Deep action-recognition methods represent video as still images, spatiotemporal volumes, encoded outputs, or recurrent input sequences.
  • The proposed approach falls into the image-based deep-learning category, enabling depth action recognition with ConvNet-compatible representations.

III. PROPOSED METHOD

The method segments continuous depth sequences, constructs dynamic images through hierarchical bidirectional rank pooling, and feeds them to ConvNets for classification. The pooling captures higher-order, nonlinear temporal dynamics at multiple scales while reducing the conventional method’s temporal bias.

  • III. PROPOSED METHOD: The framework sequentially performs action segmentation, dynamic-image construction, ConvNet training, and score fusion for classification.Action boundaries are detected using quantity of movement before three dynamic-image sets are generated and classified.
  • B. Construction of Dynamic Images: The conventional rank-pooling approach uses one temporal scale and biases the pooled feature toward early frames.These limitations motivate recursive hierarchical pooling and bidirectional processing.
  • B. Construction of Dynamic Images: Hierarchical rank pooling recursively applies rank pooling over sliding windows to capture high-order and nonlinear depth-sequence dynamics.Window size and stride determine the subsequences processed at each layer.
  • B. Construction of Dynamic Images: Bidirectional rank pooling processes depth sequences in forward and reverse temporal order, producing paired dynamic images.The combined design captures sequence dynamics at different temporal scales and in both directions.
  • B. Construction of Dynamic Images: For most relatively short actions, two layers of bidirectional rank pooling are empirically sufficient.The paper illustrates a two-layer configuration with window size three and stride one.

1) Construction of DDI:

DDIs are generated by applying hierarchical bidirectional rank pooling directly to segmented depth-map pixels. They primarily encode posture evolution, while potentially losing motion information because depth pixels are insensitive to motion.

  • 1) Construction of DDI:: Hierarchical bidirectional rank pooling directly transforms segmented depth maps into forward and backward Dynamic Depth Images.The resulting DDIs have the same spatial form as the depth maps and encode sequence evolution.
  • 1) Construction of DDI:: DDIs effectively capture posture information resembling key poses but may lose motion information because depth pixels are insensitive to motion.

2) Construction of DDNI:

DDNIs represent depth-sequence structure through surface-normal channels and hierarchical bidirectional rank pooling. Background pixels are removed using a depth-histogram threshold before normal-image construction.

  • 2) Construction of DDNI:: DDNIs are built from three surface-normal channels, then processed with hierarchical bidirectional rank pooling in forward and backward directions.The channels encode the normal components (Nx, Ny, Nz) derived from per-pixel surface normals.
  • 2) Construction of DDNI:: Background pixels beyond a threshold based on the last depth-histogram peak are reset to zero before DDNIs are calculated.A fixed tolerance is subtracted from the last peak to define the removal threshold.
  • 2) Construction of DDNI:: Figure 4 provides forward and backward DDNI examples alongside other dynamic-image representations for one gesture.

3) Construction of DDMNI:

DDMNIs isolate moving foreground regions with a Gaussian mixture model, construct normal images from those regions, and apply hierarchical bidirectional rank pooling. The resulting representations are fused through six ConvNets for recognition.

  • 3) Construction of DDMNI:: DDMNIs apply hierarchical bidirectional rank pooling to normal images constructed from GMM-detected moving foreground regions.The forward and backward DDMNIs are intended to capture motion information specifically.
  • 3) Construction of DDMNI:: Each depth sequence produces six dynamic images, which are assigned to six separately trained VGG-16 ConvNets.The six channels comprise forward and backward versions of DDI, DDNI, and DDMNI.
  • D. Score Fusion for Classification: The recognition system multiplies paired ConvNet score vectors, normalizes each representation’s scores, and multiplies the three normalized vectors element-wise.The class with the maximum final score is assigned as the recognized class.
  • 3) Construction of DDMNI:: The evaluation covers ChaLearn isolated and continuous gesture datasets and NTU RGB+D, with continuous videos segmented before recognition.

A. Settings

The experiments tune segmentation, hierarchical rank-pooling, CNN, and classifier choices. They select two-layer pooling with fixed window settings and VGG-16 for computational efficiency.

  • Segmentation: The sliding-window size was tied to the average action length L because window size theoretically affects continuous-action segmentation.Experiments used the DDI forward channel on the ChaLearn LAP ConGD validation subset.
  • Hierarchical rank pooling: Two-layer hierarchical bidirectional rank pooling with window size M_l = 3 and stride S_l = 1 was adopted for all experiments.Performance was reported as insensitive to the tested pooling parameters on the ChaLearn LAP IsoGD validation set.
  • CNN selection: 36.96% validation accuracy made VGG-19 marginally better than VGG-16, but VGG-16 was selected because their performance was comparable and its computational cost was lower.AlexNet, GoogleNet, and MobileNetV2 achieved 32.22%, 36.11%, and 34.56%, respectively.
  • Classifier selection: VGG-16 performed favorably against kNN classifiers and convolutional-layer feature variants on the ChaLearn LAP IsoGD DDI forward channel.The comparison tested k = 1, 5, 15, and 25, linear SVM, and full VGG-16.

1) Description:

The evaluation covers the ChaLearn LAP IsoGD and ConGD datasets using depth-only recognition protocols. Results compare dynamic-image channels, pooling variants, and prior methods, with DDI outperforming DDNI and DDMNI in the reported IsoGD channel comparison.

  • Dataset: The ChaLearn LAP IsoGD dataset contains 47,933 RGB-D depth sequences covering 249 gestures performed by 21 individuals.Only depth maps were used in this evaluation.
  • Evaluation protocol: IsoGD recognition rate is computed from predicted and ground-truth labels across n samples.The train, validation, and test sets contain different subjects to prevent subject overlap across splits.
  • Channel comparison: DDI achieved much better results than DDNI and DDMNI on the ChaLearn LAP IsoGD validation-set channel comparison.The paper attributes weaker normal-based channels to normalized, compressed depth data that distort structure and add artifacts.
  • Pooling comparison: Bidirectional rank pooling extracted more useful information than one-way pooling, while score fusion largely improved accuracy.Hierarchical rank pooling also encoded depth-sequence dynamics better than conventional rank pooling.
  • Visual comparison: The proposed dynamic images visually show cleaner motion information than the compared SFAM representations, which exhibit blocky artifacts.The comparison is presented in Fig. 5.

C. ChaLearn LAP ConGD Dataset

On ChaLearn LAP ConGD, the method evaluates depth-only continuous gesture recognition using the dataset’s organizer-defined splits and mean Jaccard Index, comparing its results with prior approaches.

  • ChaLearn LAP ConGD contains 47,933 RGB-D gesture instances across 22,535 videos, covering 249 gestures performed by 21 individuals.
  • Mean Jaccard Index evaluates average relative overlap between predicted and true gesture-label sequences across testing sequences.
  • The comparison includes methods combining handcrafted and deep features from RGB and depth modalities with segmentation and BoW representations.
  • The proposed method outperformed previous methods on ChaLearn LAP ConGD despite using only depth data.

D. NTU RGB+D Dataset

On NTU RGB+D, the proposed depth-only method is evaluated under cross-subject and cross-view protocols and is reported to outperform previous methods, while confusion patterns expose limitations of depth-only recognition.

  • NTU RGB+D contains more than 56,000 sequences and 4 million frames covering 60 actions performed by 40 subjects across multiple viewpoints.
  • Cross-subject training and testing use disjoint subject sets, whereas cross-view training uses cameras 2 and 3 and testing uses camera 1.
  • DDNIs outperform DDIs in cross-subject evaluation, but DDNIs and DDMNIs perform much worse in cross-view evaluation because norm vectors are sensitive to motion and viewpoint.
  • Fusing the three dynamic-image representations provides complementary shape and motion information.
  • The proposed method significantly outperformed previous skeleton-based and depth-based methods on NTU RGB+D.
  • Depth-only recognition confuses visually similar actions and human-object interactions when involved objects are difficult to distinguish in depth maps.

E. Discussions

The discussion examines cross-dataset transfer, score fusion, and feature fusion, showing limited transfer improvement and favoring product score fusion over alternatives.

  • Fine-tuning models across NTU RGB+D and ChaLearn LAP IsoGD produced no noticeable improvement in cross-dataset results.
  • Product score fusion achieved the best results among the three late score fusion methods on all three large datasets.
  • Concatenating six ConvNet feature vectors and classifying with a linear SVM achieved 40.88% on validation, below the proposed method’s 43.73%.

3) Channel fusion:

Channel-fusion experiments compare combinations of the three dynamic-depth representations and fusion costs, finding that three-way fusion performs best while requiring additional computation.

  • 3) Channel fusion:: Fusing all three representations produced the best results, while combining DDI with either DDNI or DDMNI outperformed fusing DDNI with DDMNI.
  • 3) Channel fusion:: The proposed method requires more computation than previous methods, but the reported performance improvement is considered worthwhile.
  • The three compact representations are constructed with hierarchical bidirectional rank pooling to capture spatial, temporal, and structural information at different time scales.
  • The representations allow existing ConvNet models learned from RGB video to be fine-tuned on depth data.
Loading 1804.01194v2…