Source-linked AI summary
R-CNNs for Pose Estimation and Action Detection
Georgia Gkioxari, Bharath Hariharan, Ross Girshick, Jitendra Malik
TL;DR
The paper addresses pose prediction and action classification for people in unconstrained images using task-specific R-CNN training. It evaluates the approach on PASCAL VOC, reports leading pose and action results, and introduces action detection for simultaneous localization and classification.
Problem
Pose and action recognition require predicting human keypoints or actions in unconstrained images, while standard action classification assumes ground-truth person locations at test time.
Method
The approach trains CNN-based R-CNN detectors with task-dependent losses for pose, action classification, and detection.
Results
15.2% mean AP for pose and 70.5% mAP for action classification are reported on PASCAL VOC, exceeding prior results cited by the authors.
Takeaways & Limitations
The paper extends evaluation to action detection, which jointly localizes people and classifies their actions, and releases a dataset for this task.
Abstract
from arXiv · showhide
We present convolutional neural networks for the tasks of keypoint (pose) prediction and action classification of people in unconstrained images. Our approach involves training an R-CNN detector with loss functions depending on the task being tackled. We evaluate our method on the challenging PASCAL VOC dataset and compare it to previous leading approaches. Our method gives state-of-the-art results for keypoint and action prediction. Additionally, we introduce a new dataset for action detection, the task of simultaneously localizing people and classifying their actions, and present results using our approach.
1 Introduction
The paper applies task-specific CNNs within the R-CNN framework to pose estimation and action classification, achieving leading results and introducing action detection for jointly localizing people and classifying their actions.
- The method trains convolutional neural networks within R-CNN for pose estimation and action classification.
- 15.2% mean AP on PASCAL VOC 2009 person detection validation establishes the pose result, improving over the previous 12.7% state of the art.
- 70.5% mAP on PASCAL VOC action test classification slightly exceeds the previous leading method’s 70.2%.
- Action detection removes ground-truth test-time localization by requiring simultaneous prediction of a person’s location and action.The paper frames this as applying standard object-detection evaluation to categories defined by person–action pairs.
- The introduced action-detection dataset is made available online for evaluating future methods.
2 Related work
Prior pose and action-recognition systems commonly use part detectors, geometric body models, poselets, or hand-engineered features, whereas this work builds on CNN-based R-CNN representations.
- Recent CNN work showed strong image-classification gains and broadly useful transferable features for object detection and other tasks.
- Earlier pose-estimation approaches used part detectors and geometric constraints, including pictorial structures with tree-structured body-part graphs.
- Action-classification methods commonly detect body parts or poselets and classify their activations or aligned descriptors with SVMs.
3 A Single Convolutional Neural Network for Multiple Tasks
A single multitask CNN uses separate loss functions for person detection, pose estimation, and action classification, with task-specific targets and overlap rules.
- The multitask CNN associates each task with its own loss and combines task outputs in one jointly trained network.The total loss can be a weighted sum of individual task losses, with weights reflecting task importance.
- Person detection: Person detection predicts a two-class probability vector from proposal regions labeled by overlap with ground-truth people.
- Pose estimation: Pose estimation predicts normalized body-keypoint locations using mean squared error while ignoring keypoints that are not visible.
- Regions insufficiently overlapping ground truth receive zero loss for pose and action training.
- Action classification: Action classification labels proposal regions only when overlap exceeds 0.7 and predicts a probability for each action.The higher overlap threshold is intended to ensure training regions are significantly close to the ground truth.
4 Experiments
Experiments evaluate R-CNN variants for pose estimation, action classification, and action detection on PASCAL VOC, including jointly trained networks. The results compare task-specific and multitask models using AP-based metrics.
- Pose Estimation: Pose R-CNN predicts all keypoint locations for each input region and uses one SVM classifier per keypoint to produce confidence scores.Keypoint AP is measured separately on VAL09B, and Pose R-CNN outperforms k-poselets by 19.7% relatively.
- Action Classification: Action R-CNN evaluates action labels when person locations are known, using augmented training regions overlapping ground truth by more than 0.7.This augmentation increases the training set by 100x; fc6 features work slightly better than softmax outputs and fc7 features in the reported setup.
- Action Classification: Action R-CNN performs slightly better on average than the other leading approaches reported for PASCAL VOC 2012 action classification.Table 2 reports AP (%) on the Test 2012 action-classification benchmark.
- Action Detection: Action detection removes the assumption of known person locations by requiring simultaneous localization and action classification, with correctness requiring overlap above 0.5 and the correct label.Detection-Action R-CNN is evaluated on region proposals using the standard AP metric.
- A single network for Detection, Pose and Action: A jointly trained Detection-Pose-Action R-CNN achieves 56.4% person-detection AP, 15.5% pose mean AP, and 21.6% action-detection mean AP.Its average performance is similar to networks trained separately for individual tasks.
- A single network for Detection, Pose and Action: The joint network behaves very similarly to individual networks while being N-times faster because it fine-tunes one network and processes evaluation data once.The comparison concerns multitask evaluation against separately trained task-specific networks.