Source-linked AI summary
Cross-view Action Modeling, Learning and Recognition
Jiang wang, Xiaohan Nie, Yin Xia, Ying Wu, Song-Chun Zhu
TL;DR
The paper targets the limited ability of existing action-recognition methods to recognize actions from unseen viewpoints. It proposes MST-AOG, a compositional multiview representation trained with 3D skeleton supervision but used on 2D video, and reports improved accuracy and robustness for cross-view recognition.
Problem
Existing video-based action-recognition methods are generally view-dependent, while recognition from unseen views remains insufficiently explored.
Method
MST-AOG hierarchically models geometry, appearance, and motion across actions, poses, views, body parts, and features, using 3D skeletons during training and 2D video during recognition.
Results
MST-AOG achieves 73.5% recognition accuracy on MSR-DailyActivity3D, outperforming the reported baseline methods, and significantly improves cross-view recognition accuracy and robustness overall.
Takeaways & Limitations
The representation supports recognition from unknown views without requiring 3D skeleton inputs at test time.
Takeaways & Limitations
The projection model assumes actors are sufficiently far from the camera, under which orthographic projection works well in practice.
Abstract
from arXiv · showhide
Existing methods on video-based action recognition are generally view-dependent, i.e., performing recognition from the same views seen in the training data. We present a novel multiview spatio-temporal AND-OR graph (MST-AOG) representation for cross-view action recognition, i.e., the recognition is performed on the video from an unknown and unseen view. As a compositional model, MST-AOG compactly represents the hierarchical combinatorial structures of cross-view actions by explicitly modeling the geometry, appearance and motion variations. This paper proposes effective methods to learn the structure and parameters of MST-AOG. The inference based on MST-AOG enables action recognition from novel views. The training of MST-AOG takes advantage of the 3D human skeleton data obtained from Kinect cameras to avoid annotating enormous multi-view video frames, which is error-prone and time-consuming, but the recognition does not need 3D information and is based on 2D video input. A new Multiview Action3D dataset has been created and will be released. Extensive experiments have demonstrated that this new action representation significantly improves the accuracy and robustness for cross-view action recognition on 2D videos.
1. Introduction
The paper addresses cross-view action recognition, where standard methods struggle with unseen viewpoints and view-invariant features. It introduces MST-AOG, a compositional model that represents geometry, appearance, and motion while learning view structure from 3D skeleton supervision.
- Cross-view action recognition remains underexplored because visual appearances vary substantially across viewpoints and view-invariant features are difficult to obtain.
- Enumerating views requires time-consuming annotation of videos across all viewpoints and action categories.
- MST-AOG hierarchically composes actions, poses, views, body parts, and features through conjunctive and disjunctive nodes.Its leaves represent appearance and motion features, while upper-layer grounding can capture low-resolution spatial and temporal patterns.
- The model jointly represents geometry, appearance, and motion variations across views, enabling cross-view pose detection and action classification after learning.
- Training uses Kinect-derived 3D skeletons to model projected 2D views, while recognition uses video without 3D information.Discriminative pose mining and hierarchical part sharing reduce annotation demands and support robust recognition.
2. Related Work and Our Contributions
The paper contrasts local-feature, 2D-pose, and 3D-skeleton approaches with its MST-AOG representation. MST-AOG unifies multiview geometry, appearance, and motion, using 3D skeletons for training but only 2D video for recognition.
- Related Work: Prior action-recognition methods include local-feature approaches based on spatio-temporal interest points and transfer learning for cross-view recognition.
- Related Work: 2D pose-based methods estimate or model poses for action recognition but generally were not designed specifically for cross-view actions.
- Related Work: 3D skeleton methods can recognize fine-grained actions, but this paper uses 3D skeletons only as training supervision and requires no 3D input during testing.
- Contributions: MST-AOG jointly models geometry, appearance, and motion in a compact hierarchical representation of multiple viewpoints.
- Contributions: After training, MST-AOG recognizes actions from novel views using 2D video input, with shared parts supporting effective model learning.
3. Multi-view Spatio-Temporal AOG
MST-AOG is a hierarchical compositional representation that models actions through 3D poses, projected views, body parts, and appearance and motion features. Shared 3D geometry and interpolated view-specific part models support recognition from multiple views, including low-resolution inputs.
- Model structure: MST-AOG hierarchically composes actions, poses, views, body parts, and features using AND, OR, and leaf nodes.Actions, views, and parts are modeled as AND nodes, while pose nodes are OR nodes.
- Pose and view nodes: A pose represents discriminative 3D configurations whose projections form alternative 2D view nodes.Each view node contains a star-shaped set of body-part nodes relative to a root part.
- Pose and view nodes: The model shares 3D part geometry across views and obtains novel-view 2D relationships by projection.Part offsets are modeled in 3D, projected under scaled orthographic views, and used for spatial regularization.
- Part nodes: Each part combines view-dependent appearance and motion scores, with interpolation across discretized view-angle bins.HOG represents appearance and HOF represents motion; interpolation enables part models for novel views.
- Action nodes: Action nodes combine discriminative pose nodes with low-resolution feature leaves for cases where detailed body-part localization is unreliable.The low-resolution features include foreground intensity histograms and bounding-box sizes.
4. Inference
MST-AOG inference scores candidate structures over locations, scales, and views using dynamic programming. Pose selection compares child view-node scores, while view-node part locations are optimized efficiently with distance transforms.
- Inference: Inference computes all node scores on novel-view videos using bottom-up and top-down dynamic programming on the tree structure.The process is analogous to sum-product and max-product inference in graphical models.
- Inference: Pose-node inference selects the highest-scoring child view node at each location and scale.Pose, view, and part states are represented by their locations and scales.
- Inference: Distance transforms efficiently maximize view-node scores over body-part locations.Inference first obtains response maps by convolving input frames and optical flow with appearance and motion templates across views.
- Spatio-temporal pooling: A three-level spatio-temporal pyramid produces a 73-dimensional vector for each response map.The representation contains 1 + 8 + 64 cells, followed by a linear action-score prediction.
5. Learning
Learning MST-AOG jointly estimates model parameters and discovers discriminative pose structure. Kinect skeleton data provides training-time 3D supervision, while clustering, mining, latent SVM optimization, and pruning build a compact shared representation.
- Learning objectives: Learning estimates appearance, motion, geometric, and classification parameters while discovering a discriminative 3D-pose dictionary for the model structure.The two learning tasks are parameter estimation and structural pose discovery.
- Parameter learning: Latent structural SVM learning optimizes part and view parameters while treating part locations and view angles as latent variables.Labeled locations and view angles initialize the latent variables, making learning more robust to training-data noise.
- Parameter learning: Pose-specific detectors use nearby positive examples, 5000 sampled negatives, and two rounds of hard-negative bootstrapping.The resulting linear action-score weights are learned with a linear SVM solver.
- Part clustering: Part configurations are obtained by clustering examples using 3D positions, motion, and visibility, then retaining sufficiently populated clusters.Spectral clustering produces candidate part items represented by average joint positions and motions.
- Pose mining: Discriminative poses combine part items with high category support and discrimination, and an Apriori-like search removes non-maximal candidates.Greedy set covering and detector-validation pruning reduce redundancy and weak candidates.
6. Experiments
Experiments evaluate MST-AOG on Multiview Action3D and MSR-DailyActivity3D under subject, view, and environment variations. The method achieves strong recognition while using only a single unknown-view video at test time, though object interactions remain difficult.
- Evaluation settings: Experiments use the Multiview Action3D and MSR-DailyActivity3D datasets, with recognition evaluated under three settings.The settings are cross-subject, cross-view, and cross-environment recognition.
- Evaluation settings: Testing uses videos from a single unknown view without skeleton information or videos from multiple views.This isolates recognition from unseen-view 2D video input.
- Multiview Action3D results: MST-AOG achieves the best performance under cross-subject, cross-view, and cross-environment settings and remains robust to viewpoint and environment changes.The comparison includes virtual views, Hankelet, Action Bank, Poselet, and view-specific deformable part models.
- Multiview Action3D results: The main cross-view confusions are “pick up with one hand” versus “pick up with two hands” and the subtle “drop trash” action.Their motion and appearance are similar, while dropping trash can involve extremely subtle movement for some subjects.
- MSR-DailyActivity3D results: MST-AOG achieves 73.5% recognition accuracy on MSR-DailyActivity3D, exceeding the reported baseline methods in the stated evaluation.Poselet achieves 23.75% accuracy, while Actionlet Ensemble achieves 85.5% but uses 3D skeleton information during testing and is not directly comparable.
- MSR-DailyActivity3D results: Recognition is strongest for pose- or motion-determined activities, while playing guitar and playing games remain challenging because they require human-object interaction modeling.The authors identify object-interaction modeling as a way to improve accuracy for these actions.
7. Conclusion
MST-AOG is a hierarchical representation for cross-view action recognition that models geometry, appearance, and motion variations across viewpoints. Experiments show improved accuracy and robustness across cross-view, cross-subject, and cross-environment settings, while action and pose detection remain future work.
- MST-AOG expresses geometry, appearance, and motion variations across multiple viewpoints through a hierarchical compositional model.
- The model uses 3D skeleton data for training but performs action recognition from 2D videos at unknown views.
- MST-AOG significantly improves accuracy and robustness for cross-view, cross-subject, and cross-environment action recognition.
- Detecting the views and locations of actions and poses is identified as future work.