Source-linked AI summary
Spatio-Temporal Covariance Descriptors for Action and Gesture Recognition
Andres Sanin, Conrad Sanderson, Mehrtash T. Harandi, Brian C. Lovell
TL;DR
Action and gesture recognition needs representations that are robust to viewpoint, noise, occlusions, and unreliable localisation while preserving the geometry of covariance descriptors. The paper introduces Cov3D with weighted Riemannian projection and boosting, reports superior performance across three benchmark datasets, and avoids additional video preprocessing.
Problem
Existing global representations are sensitive to viewpoint, noise, occlusions, and region-of-interest localisation, while covariance descriptors lie on a nonlinear manifold rather than a vector space.
Method
The method combines spatio-temporal Cov3D descriptors, integral-video computation, sample-weighted Riemannian locality preserving projection, and boosting to select useful spatio-temporal regions.
Results
The Cov3D-based WRLPP boosting method performs better than several recent approaches on three benchmark datasets and exceeds 90% recognition accuracy for both action and gesture recognition.
Takeaways & Limitations
The combined representation and classifier provides a state-of-the-art approach for action and gesture recognition without foreground detection, interest-point detection, or tracking.
Takeaways & Limitations
Experiments generally use no input preprocessing and evaluate recognition with 5-fold cross validation, limiting the reported scope to that evaluation setting.
Abstract
from arXiv · showhide
We propose a new action and gesture recognition method based on spatio-temporal covariance descriptors and a weighted Riemannian locality preserving projection approach that takes into account the curved space formed by the descriptors. The weighted projection is then exploited during boosting to create a final multiclass classification algorithm that employs the most useful spatio-temporal regions. We also show how the descriptors can be computed quickly through the use of integral video representations. Experiments on the UCF sport, CK+ facial expression and Cambridge hand gesture datasets indicate superior performance of the proposed method compared to several recent state-of-the-art techniques. The proposed method is robust and does not require additional processing of the videos, such as foreground detection, interest-point detection or tracking.
1. Introduction
The paper develops spatio-temporal covariance descriptors for action and gesture recognition, addressing limitations of global representations while avoiding reliance on segmentation, tracking, or interest-point detection. It combines manifold-aware weighted projection with boosting to select useful spatio-temporal regions efficiently.
- Representation approaches: Global representations encode whole images but are sensitive to viewpoint, noise, occlusions, and unreliable region-of-interest localisation.These limitations can lead to unreliable classification.
- Representation approaches: Local representations describe visual information as collections of patches, but usually require increased computation.Dense sampling can outperform interest-point approaches, while patches may have arbitrary positions and sizes.
- Covariance descriptors: Covariance descriptors provide low-dimensional representations that support efficient comparison and have improved detection performance and robustness over related descriptors such as HOG.The paper extends flat image-region covariance descriptors into the spatio-temporal setting.
- Proposed representation: Cov3D descriptors represent feature vectors from spatio-temporal windows as covariance matrices on a nonlinear Riemannian manifold rather than a vector space.The manifold structure motivates using Riemannian locality preserving projection instead of ignoring descriptor geometry.
- Proposed classifier: WRLPP incorporates training-sample weights into manifold projection, while boosting searches the many candidate windows for the most useful descriptors.The resulting approach focuses the representation on critical training samples and selected spatio-temporal regions.
- Evaluation and efficiency: Integral video representations enable fast Cov3D computation inside spatio-temporal windows, and experiments compare the method with recent approaches on three benchmark datasets.The paper evaluates action and gesture recognition performance across multiple datasets.
2. Cov3D Descriptors
Cov3D represents spatio-temporal feature windows with covariance matrices and accelerates their computation using integral videos. The descriptors combine gradient and optical-flow features, are normalized against the full video, and can be selected from many candidate regions.
- Cov3D Descriptors: Cov3D represents each spatio-temporal window by the covariance matrix of its d-dimensional feature vectors.The covariance is computed from the vectors inside region R and their mean.
- Cov3D Descriptors: Covariance descriptors provide size-independent, noise-reducing representations that fuse correlated features.These advantages arise from representing a region through covariance rather than retaining all individual feature vectors.
- Fast computation: Integral videos compute spatio-temporal covariance windows efficiently by storing first- and second-order feature sums.The approach uses d + d^2 integral videos and computes each window covariance in O(d^2) time.
- Features and regions: The feature mapping combines spatial coordinates, intensity gradients, and optical-flow quantities into a 15-dimensional feature video.The resulting covariance descriptor is a 15 × 15 matrix; optical-flow features include flow components, temporal derivatives, divergence, and vorticity.
- Features and regions: Recognition uses multiple spatio-temporal windows sampled with minimum size and interval constraints because the candidate window set is very large.The paper illustrates multiple Cov3D windows for face-expression recognition.
- Features and regions: Each regional descriptor is normalized using the diagonal of the covariance descriptor for the full feature video to improve robustness against illumination variations.The normalization retains the full-video covariance diagonal and zeroes its off-diagonal entries.
3. Classification of Actions and Gestures
The method maps spatio-temporal covariance descriptors from their Riemannian manifold into Euclidean spaces using weighted locality preservation, then selects useful regions through boosting for multiclass recognition.
- Manifold representation: Cov3D descriptors are symmetric positive definite matrices that form a connected Riemannian manifold rather than a vector space.Classification therefore uses mappings into Euclidean representations before applying conventional learning methods.
- Weighted RLPP: WRLPP weights the adjacency graph so edges involving critical training samples receive greater importance during projection.The weighted graph preserves the local geometry around samples with higher boosting weights.
- Boosting classifier: Boosting searches the large set of spatio-temporal windows and selects descriptors that best separate the classes.At each iteration, candidate descriptors are projected with the current WRLPP model and evaluated by weighted regression.
- Weighted RLPP: WRLPP maps each Cov3D matrix C to Euclidean coordinates using a learned projection matrix and kernel similarities to the training points.The mapping is defined as WRLPP(C) = A^T K_C.
- Boosting classifier: The final multiclass classifier combines one-vs-one LogitBoost classifiers trained for every unique pair of classes.Each weak learner uses projected Cov3D descriptors, while probability margins control the number of weak classifiers to reduce overfitting.
4. Experiments
Experiments evaluate mapping choices and recognition performance on sport actions, facial expressions, and hand gestures using three benchmark datasets. WRLPP gives the best mapping performance, while Cov3D-based recognition achieves the highest reported results across the evaluated datasets.
- Mapping comparison: The experiments compare six mappings from symmetric positive definite matrices to Euclidean space, including vectorisation, tangent-space methods, RLPP, and WRLPP.The mapping comparison uses detection error trade-off curves for conflicting one-vs-one classifiers on Cambridge hand gestures.
- Mapping comparison: WRLPP achieves the best overall mapping performance because it accounts for sample weights on critical training points during boosting.Original RLPP performs worse in this setting despite generally improving manifold representations over tangent-based approaches.
- UCF sport actions: The Cov3D-based approach achieves the highest accuracy on the UCF sport action dataset among the compared methods.The comparison includes HOG3D, HDN, and AFMKL.
- CK+ facial expressions: 92.3% average recognition accuracy is achieved by Cov3D on CK+, compared with 87.92% for the next-best TMS method.The average is computed over seven facial-expression classes.
- Cambridge hand gestures: The Cov3D-based approach obtains the highest performance on all four reported Cambridge hand-gesture test sets.The evaluation follows the dataset’s prescribed protocol, training on normal illumination and testing on the remaining sequences.
5. Conclusion
The paper combines Cov3D descriptors with WRLPP-based boosting for action and gesture recognition, while exploiting the Riemannian structure of the descriptors. The resulting method is reported as robust, accurate across both task types, and computationally efficient through integral video representations.
- Cov3D descriptors are extended to spatio-temporal data and computed efficiently using integral video representations.The descriptors are extracted from spatio-temporal windows, and integral video representations enable fast calculation.
- WRLPP incorporates training-sample weights into manifold projection during boosting to improve neighborhoods around critical samples.Boosting searches the large set of possible spatio-temporal windows while weighted projection emphasizes critical training samples at each iteration.
- Combining Cov3D with WRLPP boosting produces a state-of-the-art action and gesture recognition method.The method is reported to outperform several recent approaches on three benchmark datasets and achieve over 90% recognition accuracy for both action and gesture recognition.
- The method is robust and does not require foreground detection, interest-point detection, or tracking.
- The authors identify anomaly detection in surveillance videos as a related future application, especially where positive examples are scarce.