Source-linked AI summary
HOPC: Histogram of Oriented Principal Components of 3D Pointclouds for Action Recognition
Hossein Rahmani, Arif Mahmood, Du Q. Huynh, Ajmal Mian
TL;DR
3D action recognition remains vulnerable to viewpoint changes, which alter depth-image features. The paper processes pointclouds with HOPC and STKP detection in adaptive spatio-temporal supports, reporting stronger benchmark performance and viewpoint robustness.
Problem
Viewpoint variations make existing depth-based 3D action recognition methods unreliable, especially when training and testing views differ substantially.
Method
The paper directly processes 3D pointclouds using HOPC descriptors within adaptive spatio-temporal supports and an STKP detector.
Results
The proposed descriptor and detector outperform state-of-the-art methods on three benchmark datasets and show robustness to viewpoint variations on a new multiview dataset.
Takeaways & Limitations
Pointcloud-based HOPC and STKP processing provides a reported alternative for action recognition under viewpoint, noise, and speed variation.
Takeaways & Limitations
The method does not include discriminative-feature mining steps that could improve descriptor accuracy.
Abstract
from arXiv · showhide
Existing techniques for 3D action recognition are sensitive to viewpoint variations because they extract features from depth images which change significantly with viewpoint. In contrast, we directly process the pointclouds and propose a new technique for action recognition which is more robust to noise, action speed and viewpoint variations. Our technique consists of a novel descriptor and keypoint detection algorithm. The proposed descriptor is extracted at a point by encoding the Histogram of Oriented Principal Components (HOPC) within an adaptive spatio-temporal support volume around that point. Based on this descriptor, we present a novel method to detect Spatio-Temporal Key-Points (STKPs) in 3D pointcloud sequences. Experimental results show that the proposed descriptor and STKP detector outperform state-of-the-art algorithms on three benchmark human activity datasets. We also introduce a new multiview public dataset and show the robustness of our proposed method to viewpoint variations.
1 Introduction
The paper addresses viewpoint variation in 3D action recognition by directly processing pointclouds with HOPC descriptors, STKP detection, and speed normalization.
- Viewpoint variations, occlusion, and sensor noise remain major challenges for depth-camera action recognition.
- Existing methods often treat depth sequences like color videos, although depth data supports geometric descriptors and differential operations can amplify noise.
- HOPC captures local geometric characteristics by projecting PCA eigenvectors onto polyhedron directions and scaling them by eigenvalues.
- The method uses HOPC both as a holistic cell descriptor and locally at ranked spatio-temporal keypoints.
- The contributions include a HOPC descriptor, an STKP detector, a view-invariant descriptor, and action-speed normalization.
- A new public dataset contains 30 actions with subject-scale and viewpoint variations, and experiments compare the method with eight state-of-the-art methods across four datasets.
2 Related Work
Related work spans RGB, skeleton, depth-map, and trajectory approaches, but these methods face limitations from unavailable joints, noise, derivatives, and action-speed variation.
- RGB-based approaches seek view-independent representations through spatio-temporal features, geometric transformations, or latent spaces.
- Skeleton methods use 3D joint positions, but MoCap systems are expensive and their methods may not suit pointclouds with different motion-data quality.
- Joint-based representations can compromise accuracy because 3D joints do not capture all discriminative action information.
- Depth-map methods include holistic and local approaches, while derivative-based descriptors can suppress information at strong edges and silhouettes.
- Noise can reduce the efficiency of color-based interest-point detectors when extended to depth sequences, and some detectors fail for faster action execution.
- The proposed STKP detector is reported as robust to action speed, viewpoint, and actor scale, while HOPC is more noise-robust and does not require skeleton data.
3 Histogram of Oriented Principal Component (HOPC)
HOPC describes local 3D pointcloud geometry and motion by analyzing principal components within an adaptive spatio-temporal support. It resolves eigenvector-direction ambiguity, quantizes projected components, and concatenates their histograms into a noise-robust point descriptor.
- HOPC applies PCA to the scatter matrix of points within an adaptive spatio-temporal support around each point.The support is formed from accumulated pointclouds over a temporal interval and spatially bounded by a sphere.
- Each of three principal eigenvectors is projected onto directions defined by a regular icosahedron to form orientation histograms.The icosahedron supplies 20 facet centers, which correspond to histogram bins.
- HOPC resolves the 180° ambiguity of PCA eigenvectors using the distribution of vector directions and magnitudes within the support volume.The signs are selected from inner products between each eigenvector and vectors from the support.
- Squared projections suppress small, potentially noisy contributions before eigenvector projections are quantized into histogram bins.Quantization uses a threshold derived from projections of neighboring icosahedron directions.
- The three component histograms are ordered by decreasing eigenvalue and concatenated into the descriptor of point p.Each projected histogram is scaled by its corresponding eigenvalue before concatenation.
- Because the smallest principal component estimates the surface normal, HOPC encodes shape and motion while improving robustness to noise over gradient-based normals.The descriptor captures both types of information within the local spatio-temporal support volume.
4 Action Recognition
The paper develops holistic and local HOPC-based action-recognition approaches, including discriminative point pruning, STKP detection, view-invariant alignment, and bag-of-words classification.
- 4 Action Recognition: Holistic HOPC is suited to occlusion and stationary subjects, whereas local HOPC is more suitable for cross-view recognition and subjects changing spatial locations.The local descriptors are represented with bag-of-words codewords clustered by K-means and classified using an SVM with a histogram-intersection kernel.
- 4.1 Action Recognition with Holistic HOPC: The holistic approach divides sequences into spatio-temporal cells, accumulates and normalizes HOPC descriptors, concatenates cell descriptors, and uses SVM classification.The sequence is partitioned along X, Y, and T dimensions; cell descriptors are formed by summing point descriptors and applying L2 normalization.
- 4.1 Action Recognition with Holistic HOPC: Eigenratio thresholds prune ambiguous principal components caused by symmetrical surfaces and noise before constructing discriminative HOPC descriptors.Eigenvectors are retained according to whether δ12 and δ23 exceed θ; points failing both criteria are discarded.
- 4.2 STKP: Spatio-Temporal Key-Point Detection: STKP candidates are obtained by comparing spatial and spatio-temporal eigenvalue ratios, then ranked by a HOPC quality factor and locally non-maximum-suppressed.The detector removes lower-ranked keypoints within a spatial radius r′ and temporal interval 2τ′+1 around each selected point.
- 4.3 View-Invariant Key-Point Descriptor: The local approach aligns each STKP’s support volume to spatial principal axes and extracts a descriptor from a sampled 3D hyper-surface in space-time.Eigenvector sign disambiguation is used so features extracted from the aligned support volume are view invariant.
5 Adaptive Support Volume
The adaptive support-volume method addresses scale and action-speed variation by selecting spatial and temporal supports automatically, while STKPs are illustrated across multiple views.
- 5 Adaptive Support Volume: Fixed spatial and temporal supports are suboptimal because subjects differ in scale and actions differ in speed; larger supports also increase sensitivity to occlusion and action speed.The support-size trade-off motivates adaptive spatial and temporal scale selection.
- 5 Adaptive Support Volume: Figure 3 shows STKPs concentrated where movement occurs for the holding-head action across three views and presents multiview samples from the UWA3D dataset.The dataset includes pointclouds captured from different viewpoints, supporting evaluation of viewpoint variation.
- 5 Adaptive Support Volume: Automatic spatial scale selection chooses rb where the ratio of the first two spatial eigenvalues reaches a local maximum.The method achieves the same accuracy as fixed scale when each subject’s height is available.
- 5 Adaptive Support Volume: Automatic temporal scale selection is proposed to make the descriptor robust to action-speed variations.The method evaluates temporal supports beginning with τ = 1 and selects the τ associated with a local minimum of Ap; points without a local minimum are excluded as candidate keypoints.
6 Experiments
Experiments evaluate HOPC and its keypoint-based variants on three benchmark datasets and the UWA3D Multiview Activity dataset, covering same-view, cross-view, and action-speed variation settings.
- Datasets: The evaluation uses MSRAction3D, MSRGesture3D, ActionPairs3D, and the newly developed UWA3D Multiview Activity dataset.The UWA3D dataset contains 30 daily activities performed by ten subjects at different scales and viewpoints.
- MSRAction3D: On MSRAction3D, the local method with automatic spatial and temporal scale selection achieves 90.90% accuracy.Training uses subjects {1,3,5,7,9}, with the remaining subjects used for testing.
- MSRGesture3D: On MSRGesture3D, both method variants outperform prior techniques, with an average accuracy of 96.23%, 3.5% above the nearest competitor HDG.The local method with automatic scale selection obtains 93.61% accuracy.
- ActionPairs3D: On ActionPairs3D, the proposed algorithms outperform existing techniques by 2.23%, while the local method with automatic scale selection reaches 98.89% accuracy.The dataset contains six pairs of actions with similar motion and shape, and experiments use 252 folds.
- UWA3D Multiview Activity: On UWA3D same-view recognition, the methods achieve a maximum of 84.93% accuracy, while automatic temporal scale selection reaches 84.64% and 84.70% under swapped frame-rate settings.The holistic approach is used for same-view recognition; the local approach is used for cross-view recognition.
- UWA3D Multiview Activity: For UWA3D cross-view recognition, STKP detection combined with HOPC achieves 82.23% average accuracy across side views, compared with 76.56% using raw fitted hyper-surface values.Training uses front views and testing uses four side views at ±25° and ±50°.
7 Conclusion
The proposed HOPC descriptor and keypoint detector form a 3D pointcloud action-recognition technique designed to improve robustness to action speed and viewpoint variations. Evaluation covered three benchmark datasets and introduced a new multiview public dataset.
- The technique combines the HOPC descriptor with a dedicated keypoint detector for 3D action recognition.
- The proposed method is more robust to action speed and viewpoint variations than current techniques’ reported behavior.
- Evaluation was conducted on three benchmark activity-recognition datasets.
- The authors introduced a new multiview public dataset and demonstrated robustness to viewpoint variations.