Source-linked AI summary
P-CNN: Pose-based CNN Features for Action Recognition
Guilhem Chéron, Ivan Laptev, Cordelia Schmid
TL;DR
Fine-grained action recognition can require spatial and temporal structure beyond global local-feature statistics. P-CNN uses pose-aligned body-part CNN descriptors for appearance and motion, and it consistently improves over existing pose-based features across JHMDB and MPII Cooking Activities.
Problem
Global local-feature statistics may lack the spatial and temporal structure needed for fine-grained action recognition.
Method
P-CNN extracts appearance and optical-flow CNN descriptors from pose-defined body parts and aggregates them over time.
Results
P-CNN consistently outperforms HLPF on JHMDB and MPII Cooking Activities, while complementing dense trajectory features to improve state-of-the-art recognition.
Takeaways & Limitations
Pose-aligned appearance and motion features provide a complementary action representation and remain significantly more robust to pose-estimation errors than HLPF.
Takeaways & Limitations
P-CNN captures motion only between consecutive frames, so actions whose dynamics are better represented by broader temporal trajectories can favor IDT-FV.
Abstract
from arXiv · showhide
This work targets human action recognition in video. While recent methods typically represent actions by statistics of local video features, here we argue for the importance of a representation derived from human pose. To this end we propose a new Pose-based Convolutional Neural Network descriptor (P-CNN) for action recognition. The descriptor aggregates motion and appearance information along tracks of human body parts. We investigate different schemes of temporal aggregation and experiment with P-CNN features obtained both for automatically estimated and manually annotated human poses. We evaluate our method on the recent and challenging JHMDB and MPII Cooking datasets. For both datasets our method shows consistent improvement over the state of the art.
1. Introduction
Action recognition remains difficult because people, scenes, motion, and appearance vary, while fine-grained actions require stronger spatial and temporal structure. P-CNN addresses this by aligning CNN-based appearance and motion descriptors with tracked human body parts and evaluating them on two challenging datasets.
- Action recognition is difficult because motion, appearance, people, scenes, and fine-grained action differences vary substantially.
- Global statistics of local motion descriptors can lack structure needed to distinguish subtle actions such as golf swings and cooking activities.
- P-CNN uses human-pose tracks to extract separate appearance and motion CNN features for each body part.
- The descriptor investigates temporal aggregation schemes and combines normalized features across body parts, aggregation schemes, and modalities.
- P-CNN is evaluated using automatically estimated and manually annotated poses on JHMDB and MPII Cooking Activities.
- P-CNN consistently outperforms HLPF on both datasets, while combining it with dense trajectory features improves the state of the art.
2. Related work
Prior action-recognition methods largely aggregate local video descriptors globally, achieving strong coarse-action performance but potentially missing fine-grained spatio-temporal structure. P-CNN extends pose-based representations with CNN descriptors for localized body-part appearance and motion, complementing dense trajectory methods.
- Dense Trajectory features encoded with Fisher Vectors provide a strong baseline with outstanding results on challenging action-recognition benchmarks.
- CNNs had advanced image classification, but their application to video action recognition had produced limited improvements, motivating localized human-part descriptors.
- Global aggregation offers invariance to video variations but may fail to capture important spatio-temporal structure for fine-grained recognition.
- Earlier pose-based work models spatial and temporal relations of body joints, whereas P-CNN adds CNN-based body-part appearance and motion descriptors.
- The method uses temporally consistent body-joint tracks derived from video pose estimation despite imperfect pose estimates.
3. P-CNN: Pose-based CNN features
P-CNN defines pose-centered image regions and represents each body part with appearance and optical-flow CNN descriptors. Static features and temporal differences are min/max aggregated, normalized, and concatenated into a fixed-dimensional descriptor.
- P-CNN defines informative image regions from body-joint positions and extracts appearance-based and motion-based CNN descriptors per frame.
- RGB and flow patches are cropped for five regions—right hand, left hand, upper body, full body, and full image—and resized to 224 × 224 pixels.
- Each part-frame patch produces a 4096-value descriptor from the second fully connected layer of its corresponding CNN.
- Static descriptors use per-dimension minimum and maximum values over video frames, while dynamic descriptors use temporal differences with Δt = 4 frames.
- Min/max aggregation is applied to dynamic differences, preserving largest negative and positive changes; static and dynamic descriptors can be concatenated.
- The final P-CNN vector normalizes and concatenates motion and appearance descriptors across parts and aggregation schemes, yielding 160K dimensions.
4. State-of-the-art methods
The experiments compare P-CNN against high-level pose features and improved dense trajectories, using automatically tracked poses and established pose and trajectory representations. These baselines cover joint geometry, joint dynamics, and local spatio-temporal appearance and motion.
- The experimental comparisons include HLPF and improved dense trajectories alongside the proposed P-CNN features.
- Video poses are estimated per frame and linked over time with dynamic programming, enforcing estimator confidence and optical-flow consistency.
- HLPF normalizes joint positions, computes relative offsets and pairwise geometric relations, and adds dynamic trajectory features.
- HLPF quantizes feature dimensions with separate k-means codebooks and represents each video as histograms classified using an SVM with a χ2-kernel.
- Improved dense trajectories sample points densely, track them with optical flow, and compute HOG, HOF, and MBH descriptors in aligned spatio-temporal volumes.
- Fisher Vector encoding models first- and second-order statistics relative to a Gaussian mixture model and is used with a linear SVM.
5. Datasets
The evaluation uses JHMDB and MPII Cooking Activities, along with subsets targeting fully visible bodies and visually similar cooking actions. The datasets differ in action granularity, pose annotation, and evaluation metrics.
- Experiments use JHMDB, MPII Cooking Activities, and the sub-JHMDB and sub-MPII Cooking subsets.
- JHMDB: JHMDB contains 21 actions, 928 clips, and pose annotations for 31,838 frames, with accuracy averaged over three train/test splits.
- JHMDB: Sub-JHMDB contains 316 clips across 12 actions where the human body is fully visible, evaluated with accuracy over three splits.
- MPII Cooking Activities: MPII Cooking Activities contains 64 fine-grained actions plus background, 5,609 clips, and seven splits evaluated using mean Average Precision.
- Evaluation: Table 1 reports max-aggregated appearance and flow P-CNN performance as JHMDB-GT accuracy and MPII Cooking Activities-Pose mAP.
- MPII Cooking Activities: Sub-MPII Cooking focuses on wash hands and wash objects, comprising 194 clips and pose annotations for 29,997 frames.
6. Experimental results
Experiments evaluate P-CNN across feature combinations, temporal aggregation schemes, pose-quality settings, and comparisons with HLPF and IDT-FV on JHMDB and MPII Cooking. P-CNN benefits from combining body parts, appearance, optical flow, and dynamic features, remains robust to pose errors, and complements IDT-FV.
- 6.1. Performance of human part features: Combining all human parts with the full-image descriptor improves performance significantly on both datasets, while removing any part reduces performance.The selected representation uses all pose parts together with the full image.
- 6.1. Performance of human part features: Flow descriptors consistently outperform appearance descriptors, and combining appearance with flow further improves performance across body parts and their combination.The combined appearance-and-flow representation is used in the remaining evaluation.
- 6.2. Aggregating P-CNN features: 74.6% accuracy on JHMDB-GT is achieved by All-(Static+Dyn)(Max/Min-aggr) for App + OF, improving over Max-aggr by 1.2%.On MPII Cooking-Pose, the same version achieves 62.3% mAP, improving over max-aggregation by 1.5%.
- 6.2. Aggregating P-CNN features: Feature-level temporal aggregation outperforms score-level aggregation: All (App) reaches 60.4% with features-max-aggregation versus 56.1% with score-max-aggregation on JHMDB-GT.Adding first-order differences and min-aggregation improves results, whereas second-order differences and mean aggregation do not.
- 6.3. Robustness of pose-based features: For automatically estimated poses, P-CNN drops 5.7% on sub-JHMDB and 13.5% on JHMDB, compared with HLPF drops of 13.5% and 52.5%.P-CNN outperforms HLPF by 35.8% on JHMDB with automatically extracted poses.
- 6.3. Robustness of pose-based features: P-CNN outperforms HLPF significantly on MPII Cooking, including automatic-pose evaluation, with a +29.7% gain on the full dataset.On JHMDB-GT, P-CNN exceeds IDT-FV by 8.7%; late fusion reaches 79.5% for GT poses and 72.2% for automatic poses, while P-CNN also outperforms the state of the art on MPII Cooking.
7. Conclusion
The paper introduces P-CNN, which combines pose-aligned appearance and motion features, and shows strong complementarity with dense trajectory descriptors. Pose quality remains important, while adapting CNNs to body parts and modeling temporal evolution are proposed as future directions.
- P-CNN extracts appearance and flow information at human-pose positions and aggregates these features over video frames.
- P-CNN is more robust to human-pose estimation errors than existing pose-based features such as HLPF.
- Combining P-CNN with IDT-FV significantly improves current state-of-the-art action-recognition performance.
- Correct human-pose estimation leads to significant improvements in action recognition, underscoring pose's discriminative value.
- Future work includes fine-tuning CNNs for individual body parts and modeling frame evolution with RNNs.