Source-linked AI summary
Pose from Action: Unsupervised Learning of Pose Features based on Motion
Senthil Purushwalkam, Abhinav Gupta
TL;DR
The paper addresses how to learn pose-sensitive visual representations without human-labeled data. It uses motion patterns in action videos as supervision, jointly learning appearance and optical-flow representations through an automatically generated transformation-prediction task. The learned representation transfers across pose estimation and action-recognition benchmarks, outperforming other video-based unsupervised methods consistently across the reported tasks.
Problem
Manual labeling is costly, and it is unclear whether classification pretraining is the right general approach for learning visual features.
Method
The method jointly trains Appearance and Motion ConvNets to predict whether optical-flow motion causes the appearance change between two video frames, using automatically generated examples.
Results
The learned representation transfers to FLIC pose estimation, PASCAL static-image action recognition, and UCF101 and HMDB51 video action recognition, outperforming other video-based unsupervised methods consistently.
Takeaways & Limitations
Motion supervision from human action videos can produce pose-related appearance features that transfer across pose estimation and action-recognition tasks.
Abstract
from arXiv · showhide
Human actions are comprised of a sequence of poses. This makes videos of humans a rich and dense source of human poses. We propose an unsupervised method to learn pose features from videos that exploits a signal which is complementary to appearance and can be used as supervision: motion. The key idea is that humans go through poses in a predictable manner while performing actions. Hence, given two poses, it should be possible to model the motion that caused the change between them. We represent each of the poses as a feature in a CNN (Appearance ConvNet) and generate a motion encoding from optical flow maps using a separate CNN (Motion ConvNet). The data for this task is automatically generated allowing us to train without human supervision. We demonstrate the strength of the learned representation by finetuning the trained model for Pose Estimation on the FLIC dataset, for static image action recognition on PASCAL and for action recognition in videos on UCF101 and HMDB51.
1 Introduction
The paper questions reliance on labeled classification for visual representation learning and proposes motion patterns in videos as automatically available supervision for learning pose-sensitive appearance features.
- Manual image labeling is a major bottleneck, motivating unsupervised visual representation learning from signals that can be mined automatically.
- Existing discriminative approaches learn representations through auxiliary tasks whose ground truth can be generated automatically and may transfer through fine-tuning.
- Video-based methods have used sensors or viewpoint changes, but motion patterns provide a complementary and stronger supervisory signal.
- Because similar pose pairs are associated with similar motion patterns, the method trains appearance representations so poses linked to similar motions cluster together.
- The proposed system jointly learns appearance and motion convolutional features by predicting whether a motion transformation explains a change between two RGB appearances.
2 Related Work
Related work spans discriminative unsupervised learning, video-based temporal or transformation consistency, and action-recognition systems combining appearance, pose, and motion. The paper positions its method as learning pose features from unlabeled action videos without pose annotations.
- Discriminative unsupervised methods formulate surrogate tasks with automatically available signals, forcing networks to learn semantics useful for downstream recognition or detection.
- Video-based approaches exploit temporal coherence or cross-frame similarities, while some use viewpoint or ego-motion constraints to learn invariant or predictable representations.
- Action Recognition and Pose Estimation: Action recognition commonly combines appearance, pose, and motion features, but deep models trained directly on frame stacks can be computationally expensive.
- Action Recognition and Pose Estimation: Actions consist of pose sequences and interactions with the environment, making videos a rich source of diverse human poses.
- Action Recognition and Pose Estimation: The paper claims to be the first to learn pose features from videos with deep networks in an unsupervised fashion.
3 Approach
The approach jointly trains Appearance and Motion ConvNets to predict whether optical-flow motion explains the appearance change between two video frames. Automatically sampled matching and mismatching triplets provide supervision without human labels.
- The surrogate task predicts which transformation causes a change between two appearances, requiring appearance features that encode object properties including pose.
- The Appearance ConvNet processes RGB images while the Motion ConvNet encodes stacks of optical-flow maps, and their features are jointly learned.
- Positive examples pair appearance frames and intervening flow from the same temporal snippet, while negatives are mined from other videos to avoid repetitive-action confounds.
- Two shared-parameter Appearance ConvNets, one Motion ConvNet, and a two-layer classifier form a single system trained by standard backpropagation.
- The implementation fixes the frame separation at ∆n = 12, produces 4096-dimensional features for each stream, concatenates them into 12288 dimensions, and performs binary classification.
- The joint system reaches around 96% on the held-out binary classification task, compared with a 66% baseline with two negatives per positive.
4 Experiments
The experiments assess whether the unsupervised Appearance ConvNet learns transferable pose-related features through qualitative analysis and finetuning on pose estimation and action-recognition tasks. Results indicate useful pose structure and transfer across FLIC, UCF101, HMDB51, and PASCAL Action Classification.
- Qualitative analysis of learned models: The first convolutional layer learns gradient-like filters, providing a basic qualitative indication that the Appearance ConvNet models meaningful visual structure.The visualization covers 96 filters in the first convolutional layer.
- Qualitative analysis of learned models: Closest pairs in the Appearance ConvNet’s FC6 feature space match human poses reasonably well, suggesting that the representation captures pose properties.The qualitative analysis computes nearest images using Euclidean distance on LSP appearance features.
- Pose Estimation: Pose estimation uses a VGG-M-based network with five copied convolutional layers, deconvolution upsampling, and 1x1 convolutions that predict joint heat maps.Images are cropped around annotated humans, rescaled to 256x256, and represented by nine 60x60 joint heat maps.
- Pose Estimation: The Appearance ConvNet improves over random initialization and an unsupervised baseline on FLIC pose estimation, showing transferable pose features.Four Pose ConvNet models are compared using Strict PCP and PDJ evaluations.
- Action Recognition: The learned representation improves action-recognition accuracy over random initialization by 12.3% on UCF101 and 7.2% on HMDB51.Finetuning only the last two fully connected layers also beats training from scratch on HMDB51 and performs comparably on UCF101.
- Static Image PASCAL Action Classification: Using 50 randomly sampled images, the Appearance ConvNet improves PASCAL Action Classification performance by 2.5% over [22].The comparison uses the same limited-data setting as the baseline experiment.
5 Conclusion and Future Work
The paper presents an unsupervised motion-supervised algorithm that learns appearance representations encoding pose features from action videos. Its representation transfers across pose estimation and action-recognition tasks, outperforming two video-based unsupervised methods consistently.
- Motion in action videos supervises an appearance representation designed to learn pose features without manual labels.
- The learned representation is evaluated on pose estimation in FLIC, video action recognition on UCF101 and HMDB51, and still-image action recognition on PASCAL VOC.
- Finetuning results emphasize the high transferability of the learned representations across these tasks.
- The representation performs better consistently than two other video-based unsupervised algorithms on the evaluated tasks.
- The authors identify extending the method to generic videos as a future direction.