Source-linked AI summary
Learning a Deep Model for Human Action Recognition from Novel Viewpoints
Hossein Rahmani, Ajmal Mian, Mubarak Shah
TL;DR
Human action recognition from unknown and unseen views requires transfer across viewpoints despite nonlinear action manifolds and limited view-specific training data. The paper learns a single R-NKTM from synthetic 3D-model trajectories to map descriptors into a shared high-level space, and reports higher accuracy than existing methods across benchmark datasets. Its scope relies on assumptions about a shared high-level representation for the same action across views.
Problem
Cross-view action recognition must recognize actions from unknown and unseen viewpoints, where existing linear transfer methods may fail to capture realistic nonlinear action manifolds.
Method
R-NKTM is a single deep fully-connected nonlinear model learned from synthetic dense trajectories of realistic 3D human models fitted to mocap data, using dummy labels and shared codebook features.
Results
The method outperforms existing state-of-the-art cross-view recognition methods on benchmark datasets; on IXMAS it achieves 74.1% average recognition accuracy, about 7% above nCTE.
Takeaways & Limitations
A single R-NKTM and codebook can transfer descriptors across unknown views and allow new real-video action classes to be added without retraining the transfer model.
Takeaways & Limitations
The approach assumes that same-action videos from different views share a common high-level representation and that action- and viewpoint-related features can be separated nonlinearly.
Abstract
from arXiv · showhide
Recognizing human actions from unknown and unseen (novel) views is a challenging problem. We propose a Robust Non-Linear Knowledge Transfer Model (R-NKTM) for human action recognition from novel views. The proposed R-NKTM is a deep fully-connected neural network that transfers knowledge of human actions from any unknown view to a shared high-level virtual view by finding a non-linear virtual path that connects the views. The R-NKTM is learned from dense trajectories of synthetic 3D human models fitted to real motion capture data and generalizes to real videos of human actions. The strength of our technique is that we learn a single R-NKTM for all actions and all viewpoints for knowledge transfer of any real human action video without the need for re-training or fine-tuning the model. Thus, R-NKTM can efficiently scale to incorporate new action classes. R-NKTM is learned with dummy labels and does not require knowledge of the camera viewpoint at any stage. Experiments on three benchmark cross-view human action datasets show that our method outperforms existing state-of-the-art.
1 INTRODUCTION
Cross-view action recognition must handle unknown and unseen viewpoints, where linear transfer methods and view-specific training are limited. R-NKTM addresses this with a single nonlinear model learned from synthetic trajectories that transfers descriptors to a shared high-level space.
- Motivation: Unknown and unseen viewpoints remain difficult because existing methods cannot capture nonlinear action manifolds and may require many labeled samples or viewpoint-specific classifiers.These limitations hinder recognition when test viewpoints are unavailable during training.
- Approach: R-NKTM models cross-view recognition as nonlinear knowledge transfer from multiple views to a shared compact high-level representation.The model uses a nonlinear virtual path rather than assuming linear transformations between viewpoints.
- Approach: A single deep fully-connected R-NKTM maps descriptors from all actions and camera viewpoints, using dummy labels rather than action labels during model learning.Action labels are introduced later to train the subsequent classifier.
- Training data: Synthetic trajectories from realistic 3D human models fitted to real mocap data provide training data for the model and a general codebook shared with real videos.The synthetic corpus addresses the large data requirement of nonlinear model learning.
- Contribution: The same learned R-NKTM and codebook support adding new action classes from real videos without retraining the transfer model.The paper reports comparisons with eight existing methods across four benchmark datasets, with faster and higher-accuracy performance.
- Novelty: The revised method removes canonical-view identification and replaces earlier cylinder-based human approximations with realistic 3D models that can vary body shape and size.These changes address limitations of the authors' prior NKTM model.
2 RELATED WORK
Prior cross-view action-recognition work uses geometry, view-invariant features, transfer learning, and deep models, but often relies on restrictive assumptions or training correspondences. The reviewed methods include linear transformations, geometric representations, and learned statistical connections across views.
- Geometry-based methods: Geometry-based methods seek view invariance through 2D trajectory curvature, contour correspondences, or geometric action representations.These approaches impose geometric structure on the observed action trajectories.
- View-invariant features: Self-Similarity Matrix and Hankelet methods represent actions through frame similarities or short-tracklet dynamics, but perform poorly for viewpoints far from training views.The cited limitation is especially relevant to substantially different viewpoints such as top views.
- Transfer learning: Transfer-learning methods learn statistical connections using split-based features, bilingual visual words, or transferable dictionary pairs.Several require frame-level or video-level feature correspondence during training, limiting their applications.
- Knowledge transfer: Virtual-view methods connect source and target descriptors through sampled linear transformations along an assumed smooth path.Li and Zickler's method requires samples from both source and target views during training.
- Knowledge transfer: Wang et al. learn separate linear transformations for body parts and use exhaustive matching with an AND-OR Graph for unseen views.Their method also requires 3D skeleton training data, which is not always available.
- Deep learning: Deep learning models construct hierarchical high-level representations from low-level features and have been applied to video-based action recognition.The passage places these models within the broader development of action-recognition methods.
3 PROPOSED TECHNIQUE
The proposed technique learns R-NKTM from synthetic dense trajectories and uses it to transfer view-dependent action descriptors to a shared high-level representation. Real-video descriptors are then passed through the learned model to form cross-view action descriptors, while dummy-label training encourages view consistency.
- Feature extraction: The pipeline generates synthetic dense trajectories by fitting 3D human models to mocap sequences and projecting them from numerous viewpoints.Projected 2D trajectories are extracted from synthetic videos for model learning.
- Cross-view action description: For real videos, outputs from the learned transformation layers are concatenated into cross-view action descriptors, excluding the final fully-connected and softmax layers.The excluded layers correspond to dummy labels used during R-NKTM learning.
- Feature extraction: Dense trajectory descriptors are encoded with a bag-of-features representation using a shared codebook for synthetic and real videos.The trajectory-only representation is computationally efficient and more robust to clothing and lighting changes than appearance-based descriptors.
- R-NKTM learning: Dummy labels encourage descriptors from different views of the same mocap sequence to produce the same output class, without using action labels.The final classification layer is tied to unique labels assigned to the mocap sequences and is discarded during feature extraction.
- R-NKTM learning: The model represents each view transfer as a sequence of non-linear transformations along a virtual path connecting the input view to the shared high-level view.The formulation assumes that different views of the same action share a high-level feature representation.
4 EXPERIMENTS
Experiments evaluate R-NKTM on four benchmark datasets under unknown and unseen viewpoints, comparing it with prior action-recognition and view-transfer methods. R-NKTM generally achieves the strongest recognition results, including substantial gains for challenging top-view and previously unmatched actions, while supporting efficient addition of new classes.
- Datasets and protocol: R-NKTM is evaluated on IXMAS, UWA3DII, N-UCLA, and UCF Sports against state-of-the-art action-recognition methods.The evaluation includes unknown and unseen views without target-view videos, labels, or correspondences during training.
- IXMAS: 74.1% average accuracy on IXMAS is about 7% higher than nCTE, with R-NKTM outperforming NKTM and most state-of-the-art methods across view pairs.R-NKTM also outperforms NKTM, which requires the same canonical view for all actions.
- Class-specific results: R-NKTM achieves higher class-specific accuracy than NKTM for all IXMAS actions except check watch.On UWA3DII, jumping jack reaches 95.4% average accuracy, whereas phone answering reaches 33.3%.
- UWA3DII: 67.5% average accuracy when UWA3DII view 4 is the test view is about 8% higher than nCTE.View 4 is a challenging top view because the subject’s lower body is not fully captured.
- UWA3DII: 76.8% average accuracy on three actions absent from the CMU mocap data is about 10% higher than nCTE.The result supports generalization to real-video actions without similar synthetic training actions.
- Descriptor analysis: Adding virtual views increases recognition accuracy for every UWA3DII source-target pair, while combined appearance and motion descriptors improve most IXMAS side-view combinations.The largest incremental gain occurs when camera 4 is used as training or test view; the smallest occurs for the similar 0|1 pair.
- Generalization and computation: The same R-NKTM learned without real action labels outperforms current cross-view methods across IXMAS, UWA3DII, and N-UCLA, and adding a new class costs only SVM training.Unlike nCTE and AOG, the method does not require expensive spatio-temporal matching or retraining structured models for each new class.
5 CONCLUSION
The paper concludes that an unsupervised R-NKTM trained once on synthetic motion trajectories can generalize to real videos and support cross-view recognition. Its trajectory representation uses a shared codebook and can incorporate virtual-view features into dataset-specific descriptors.
- Conclusion: R-NKTM is learned unsupervised because its training labels are dummy labels unrelated to the actions being recognized.The model is trained once using synthetic data and is intended to generalize to real data.
- Descriptor combination: Adding HOG, HOF, and MBH features to the cross-view descriptor improves accuracy for most IXMAS source-target combinations from side views.The cited table reports comparative results for combined descriptors and cross-view-trajectory-only descriptors.
- Virtual-view features: Adding virtual-view features increases recognition accuracy for all UWA3DII source-target view pairs.The conclusion’s cited UWA3DII figure summarizes the effect of adding features from different layers to the cross-view descriptor.
- Synthetic training data: Synthetic 3D human videos are generated by fitting models to motion-capture data, projecting them across viewing directions, and extracting dense trajectories.Known correspondence between 3D poses makes the synthetic trajectories realistic and computationally tractable.
- Trajectory representation: A general k-means codebook represents synthetic trajectories and is reused to encode trajectories from real videos.The same representation supports learning the R-NKTM from synthetic data and applying it to real videos.