PaperlayerView exact revision on arXiv

Source-linked AI summary

Histogram of Oriented Principal Components for Cross-View Action Recognition

Hossein Rahmani, Arif Mahmood, Du Huynh, Ajmal Mian

arXiv:1409.6813v2cs.CV

TL;DR

Viewpoint-dependent depth representations limit 3D action recognition, especially for unknown and unseen views. The paper directly processes pointcloud sequences with HOPC, STK detection, Local HOPC, and STK-D, achieving significantly better accuracy than nine existing techniques across two multi-view and three single-view datasets.

  • Problem

    Depth-image features are viewpoint dependent, while cross-view action recognition from 3D pointcloud videos remains underexplored.

  • Method

    The paper directly processes 3D pointcloud sequences using HOPC, view-invariant STK-based Local HOPC descriptors, and the global STK-D descriptor.

  • Results

    The proposed descriptors achieved significantly better accuracy than nine existing state-of-the-art techniques across two multi-view and three single-view datasets.

  • Takeaways & Limitations

    Local HOPC combined with global STK-Distribution achieves state-of-the-art results on two standard cross-view action-recognition datasets without requiring skeleton data.

  • Takeaways & Limitations

    Skeleton-based comparison methods can be limited by unreliable or unavailable skeleton data, especially under occlusion or non-upright poses.

Abstract

from arXiv · show

Existing techniques for 3D action recognition are sensitive to viewpoint variations because they extract features from depth images which are viewpoint dependent. In contrast, we directly process pointclouds for cross-view action recognition from unknown and unseen views. We propose the Histogram of Oriented Principal Components (HOPC) descriptor that is robust to noise, viewpoint, scale and action speed variations. At a 3D point, HOPC is computed by projecting the three scaled eigenvectors of the pointcloud within its local spatio-temporal support volume onto the vertices of a regular dodecahedron. HOPC is also used for the detection of Spatio-Temporal Keypoints (STK) in 3D pointcloud sequences so that view-invariant STK descriptors (or Local HOPC descriptors) at these key locations only are used for action recognition. We also propose a global descriptor computed from the normalized spatio-temporal distribution of STKs in 4-D, which we refer to as STK-D. We have evaluated the performance of our proposed descriptors against nine existing techniques on two cross-view and three single-view human action recognition datasets. The Experimental results show that our techniques provide significant improvement over state-of-the-art methods.

1 INTRODUCTION

3D action recognition remains difficult across viewpoints because depth representations depend on the sensor view. The paper directly processes pointcloud sequences with HOPC, STK-based descriptors, and STK-D for view- and speed-invariant recognition.

  • Viewpoint variation, occlusion, sensor noise, and action-speed differences make depth-based action recognition challenging.
  • Cross-view recognition from 3D pointcloud videos is underexplored, although pointclouds may provide view-invariant geometric features.
  • The method directly processes 3D pointcloud sequences and extracts view-invariant Local HOPC descriptors from discriminative points.
  • Orientation normalization aligns local support volumes for view invariance, while automatic temporal scale selection addresses different action speeds.
  • HOPC encodes shape and motion, STK detection identifies suitable discriminative points, and STK-D describes their global spatio-temporal distribution.
  • The descriptors were evaluated on two multi-view and three single-view datasets and achieved significantly better accuracy than nine existing state-of-the-art techniques.

2 RELATED WORK

Existing action-recognition methods use color, skeleton, depth, or transferred representations, but cross-view recognition remains constrained by viewpoint, data, and detector reliability. The paper motivates direct geometric processing of pointcloud videos as an alternative.

  • Training separate classifiers for every view requires many labeled samples and becomes infeasible as the number of action categories grows.
  • Geometric-transformation approaches critically rely on accurate body-joint and contour detection, which remain open problems in real-world settings.
  • Knowledge-transfer methods can be inapplicable or perform poorly when recognition uses unknown or unseen views, and some require 3D skeleton training data.
  • Skeleton methods depend on specialized motion-capture equipment or estimated joints, while skeleton data may be unavailable or unreliable under occlusion and non-upright poses.
  • Depth-based methods use global or local features, but noise can make conventional interest-point detectors select irrelevant locations and reduce effectiveness.
  • Motion-trajectory methods are unreliable in depth sequences, and some alternative interest-point methods fail when action execution is faster than the detection threshold.

3 HOPC: HISTOGRAM OF ORIENTED PRINCIPAL COMPONENTS

HOPC represents local pointcloud geometry and motion through principal components computed in spatial or spatio-temporal support volumes. It projects oriented eigenvectors into dodecahedron-defined histogram bins and concatenates the resulting component histograms.

  • For each point, HOPC uses a spatial support volume within one frame and a spatio-temporal support volume formed by merging nearby frames.
  • The covariance matrix of each support volume is eigendecomposed into ordered eigenvalues and three corresponding orthonormal eigenvectors.
  • HOPC projects each eigenvector onto directions defined by the vertices of a regular polyhedron, using a regular dodecahedron with 20 histogram bins.
  • Eigenvector orientation ambiguity is resolved using point-vector directions and magnitudes, with squared projections suppressing small values often caused by noise.
  • Projection values are thresholded and quantized into histogram bins before the three eigenvector histograms are concatenated in decreasing eigenvalue order.
  • The spatial HOPC descriptor encodes local shape, whereas the spatio-temporal descriptor captures both shape and motion.
  • HOPC surface normals are more noise-robust than gradient-based normals, while its first two principal components encode more dominant local structure.

4 SPATIO-TEMPORAL KEYPOINT (STK) DETECTION

The STK detector selects repeatable, uniquely describable points where spatio-temporal variation is significant, then suppresses nearby candidates to retain a diverse set of keypoints.

  • STK detection targets repeatability under noise and viewpoint changes, a unique local coordinate basis, and significant spatio-temporal variation.
  • Eigenvalue-ratio constraints eliminate ambiguous points on symmetrical surfaces, where principal components are not uniquely defined.The detector uses a small noise margin ϵSTK in these conditions.
  • The quality factor ηp is low for stationary support volumes and large when significant space-time motion changes eigenvector direction and magnitude.
  • Non-maximum suppression sorts candidates by quality and discards nearby points until the desired number nk of STKs remains.The suppression neighborhood uses spatial radius r1 and temporal interval [t−τ1,t+τ1].

5 VIEW-INVARIANT STK DESCRIPTION (LOCAL HOPC)

Local HOPC descriptors are computed at STKs after aligning each local support volume to a unique eigenvector-based coordinate system, producing view-invariant spatio-temporal patterns.

  • Orientation normalization applies the eigenvectors of each STK’s spatial covariance matrix as a local object-centered coordinate basis.The resulting matrix is orthonormal and serves as a valid 3D rotation matrix.
  • The centered points in an STK’s spatio-temporal support volume are rotated into a canonical coordinate system whose principal components align with X, Y, and Z.The largest, second-largest, and smallest eigenvalues map to X, Y, and Z, respectively.
  • Eigenratio tests determine which local HOPC components contribute when principal directions are ambiguous.All eigenvectors contribute when both relevant ratios exceed θl; otherwise ambiguous components are omitted.
  • The normalized support volume is partitioned into γ spatio-temporal cells, whose accumulated descriptors are concatenated into the final view-invariant descriptor hv.
  • The resulting STK descriptors encode view-invariant spatio-temporal patterns for action description.

6 ACTION DESCRIPTION

Action descriptions combine discriminative local STK patterns from a codebook with the global 4D distribution of STKs, preserving both local and spatio-temporal positional information.

  • 6.1 Bag-of-Words Action Description: STK descriptors are clustered into a view- and action-independent codebook, quantized by nearest Euclidean distance, and classified using an SVM with a histogram intersection kernel.
  • 6.2 Mining Discriminative Codebooks: F-scores rank codewords by discrimination, enabling removal of non-discriminative histogram bins.Selecting the top 98% of 1500 features improved accuracy by up to 1.5% in the experiments.
  • 6.3 Encoding Spatio-Temporal STK Distribution: The bag-of-words representation captures local spatio-temporal information but ignores relationships among STKs, motivating a separate distribution descriptor.
  • 6.3 Encoding Spatio-Temporal STK Distribution: A 120-cell regular polychoron supplies 600 vertices that serve as uniformly distributed 4D histogram bins for STK-D.
  • 6.3 Encoding Spatio-Temporal STK Distribution: STK-D provides a compact, discriminative representation of the global spatio-temporal distribution of STKs.

7 ADAPTABLE SUPPORT VOLUME

The adaptable support volume addresses subject-scale and action-speed variation by selecting spatial and temporal scales rather than using fixed values.

  • Fixed spatial radius r and temporal scale τ are suboptimal because subjects differ in size and actions differ in execution speed.
  • Increasing r captures more shape information but increases vulnerability to occlusion, while smaller τ improves localization but may miss slow actions.
  • 7.1 Automatic Spatial Scale Selection: When subject height hs is available, the spatial scale is selected as r = σhs with 0 < σ < 1.
  • 7.2 Automatic Temporal Scale Selection: Temporal scale selection evaluates candidate scales τ = 1,…,τm and chooses the scale minimizing Ap for each point.
  • 7.2 Automatic Temporal Scale Selection: For hand waving, the selected temporal scale is τ* = 3 at slow speed, τ* = 2 at moderate speed, and τ* = 1 at fast speed.The number of frames decreases as action speed increases.
  • 7.2 Automatic Temporal Scale Selection: Temporal scale is selected independently for each STK; STKs whose optimum reaches τm are discarded.

8 EXPERIMENTS

The experiments evaluate HOPC-based descriptors across five datasets and compare them with nine existing methods. Local HOPC combined with STK-D performs particularly strongly for cross-view recognition, while parameter and scale analyses assess robustness.

  • Experimental setup: The evaluation covers two multi-view and three single-view datasets, comparing the proposed descriptors with nine existing action-recognition methods.The multi-view datasets are Northwestern-UCLA and UWA3D; the single-view datasets are MSR Action3D, MSR Daily Activity3D, and MSR Gesture3D.
  • Cross-view datasets: On Northwestern-UCLA, Local HOPC+STK-D improved performance by 8.1%, reaching 80% accuracy and about 6% above LARP.LARP requires skeleton data, whereas the proposed method does not.
  • Cross-view datasets: On UWA3D, combining Local HOPC with STK-D improved performance by 8.2%, achieving 52.2% accuracy and about 9% above LARP.The combined method achieved the highest accuracy in all training/test-view combinations except one.
  • Single-view datasets: On MSR Daily Activity3D, Holistic HOPC achieved 88.8% average accuracy, while Local HOPC+STK-D exceeded AOG, HOJ3D, and LARP but trailed HON4D and SNV.HON4D and SNV assume matching viewpoints for training and testing.
  • Single-view datasets: On MSR Gesture3D, Holistic HOPC achieved 96.2% average accuracy, while Local HOPC+STK-D achieved 93.6%, exceeding HON4D.Methods requiring 3D joint positions were excluded because this dataset does not provide them.
  • Parameter analysis: Automatic temporal scale selection achieved higher accuracy than a constant temporal scale, supporting robustness to action-speed variation.The number of STKs produced the best accuracy at nk = 400, with performance stable up to nk = 700.

9 DISCUSSION AND CONCLUSION

The paper addresses viewpoint-related degradation by processing 3D pointcloud videos directly rather than treating them as depth-image sequences. Its Local HOPC and global STK-Distribution combination achieves state-of-the-art results on two cross-view datasets without requiring skeleton data.

  • Discussion and conclusion: The proposed algorithm directly processes 3D pointcloud videos to improve robustness to viewpoint, subject-scale, and action-speed variations.The method integrates HOPC with spatio-temporal keypoint detection.
  • Discussion and conclusion: Local HOPC combined with global STK-Distribution achieves state-of-the-art results on two standard cross-view action-recognition datasets.The approach does not require skeleton data, unlike several compared methods.
  • Discussion and conclusion: The method avoids reliance on skeleton data, which can be unavailable or unreliable under partial visibility, non-upright poses, background contact, or elevated camera angles.These conditions create difficulties for joint estimation methods.
Loading 1409.6813v2…