PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextual Action Recognition with R*CNN

Georgia Gkioxari, Ross Girshick, Jitendra Malik

arXiv:1505.01197v3cs.CV

TL;DR

Action recognition must combine a person’s pose with contextual cues from objects, scenes, and other people. R*CNN adapts RCNN to select and combine multiple regions while learning action-specific representations, achieving strong action-recognition results and state-of-the-art attribute classification. Its action-recognition performance remains weak for some motion-dependent categories, motivating hybrid appearance-and-motion approaches.

  • Problem

    Action recognition from still images requires using pose, surrounding objects, interactions, and scene context rather than relying on the person alone.

  • Method

    R*CNN adapts RCNN to use a primary person region and an automatically selected contextual secondary region, with features and models learned jointly.

  • Results

    R*CNN achieves 90.2% mean AP on PASCAL VOC Actions, 26.7% mean AP on MPII Human Pose, and state-of-the-art performance on Berkeley Attributes of People.

  • Takeaways & Limitations

    The selected secondary regions capture instance-specific contextual modes for action recognition and attribute-specific body or scene parts for fine-grained classification.

  • Takeaways & Limitations

    R*CNN performs badly on several MPII categories, including Yoga, Cooking or food preparation, and Video exercise workout, where movement is needed to boost performance.

Abstract

from arXiv · show

There are multiple cues in an image which reveal what action a person is performing. For example, a jogger has a pose that is characteristic for jogging, but the scene (e.g. road, trail) and the presence of other joggers can be an additional source of information. In this work, we exploit the simple observation that actions are accompanied by contextual cues to build a strong action recognition system. We adapt RCNN to use more than one region for classification while still maintaining the ability to localize the action. We call our system R*CNN. The action-specific models and the feature maps are trained jointly, allowing for action specific representations to emerge. R*CNN achieves 90.2% mean AP on the PASAL VOC Action dataset, outperforming all other approaches in the field by a significant margin. Last, we show that R*CNN is not limited to action recognition. In particular, R*CNN can also be used to tackle fine-grained tasks such as attribute classification. We validate this claim by reporting state-of-the-art performance on the Berkeley Attributes of People dataset.

1. Introduction

R*CNN recognizes actions by combining the person’s primary region with automatically selected contextual regions. It jointly learns action-specific features and models, achieving strong results on action and attribute recognition.

  • R*CNN adapts RCNN to use a primary person region and a secondary region that automatically discovers contextual cues.The secondary region can capture information from objects, the scene, or other people relevant to the action.
  • For each action, R*CNN selects the most informative candidate secondary region and adds its score to the primary-region score.Candidate regions are supplied by region proposals, and a max operation performs the selection.
  • The feature maps and action-specific models are trained jointly, allowing action-specific representations to emerge.The method uses stochastic gradient descent and builds on the Fast RCNN implementation.
  • 90.2% mean AP on PASCAL VOC Actions, improving the previous state-of-the-art approach by 6 percentage points.On MPII Human Pose, R*CNN achieves 26.7% mean AP versus 5.5% for the best performing approach reported there.
  • R*CNN also achieves state-of-the-art performance on the Berkeley Attributes of People dataset for fine-grained attribute recognition.Its selected secondary regions focus on parts specific to the attribute class being considered.

2. Related Work

Prior action-recognition methods use holistic, pose-based, object-based, or hand-engineered contextual cues. R*CNN instead uses bottom-up region proposals and jointly learned representations to select instance-specific contextual regions.

  • Action recognition: Many existing action-recognition approaches extract features from the person bounding box and combine them with whole-image context or object models.
  • Action recognition: Prior methods include poselets, body-part alignment, region sampling, object detection scores, and human-object interaction models.
  • Action recognition: R*CNN uses bottom-up region proposals as secondary-region candidates without anchoring them to specific aspect ratios, locations, or the ground-truth box.
  • Action recognition: R*CNN jointly learns feature maps and scoring-model weights, allowing representations of human-object, human-scene, and human-human relations to emerge.
  • Scene and Context: Scene context research shows that violating object-setting relationships weakens human object recognition, motivating contextual modeling.
  • Multiple-Instance Learning: The secondary region is treated as an unknown latent variable inferred through a max operation during end-to-end SGD optimization.

3. Implementation

R*CNN extends Fast R-CNN by combining a person-centered primary region with automatically selected contextual regions. Its max-based secondary-region selection, joint training, and ROI-based processing define the implementation.

  • Region selection: The primary region is the person bounding box, while bottom-up region proposals provide candidate secondary regions filtered by overlap bounds.The implementation uses Selective Search proposals, with overlap thresholds defining which candidates are considered.
  • Scoring: The most informative secondary region is selected through a max operation, then its action score is added to the primary-region score before softmax prediction.The resulting posterior probabilities are used to predict action labels.
  • Architecture: R*CNN extends the FRCN pipeline by combining predictions from a primary person region and candidate secondary regions.Each secondary candidate predicts independently, and the candidate scores are combined using a max operation.
  • Learning: Training minimizes softmax log loss with stochastic gradient descent and backpropagation over labeled image-region examples.The true label is assigned to each training example’s primary region.
  • Learning: Training uses overlapping regions as data augmentation and randomly samples secondary candidates for each primary region.Primary regions overlapping a ground-truth box by more than 0.5 are included, and N secondary regions are sampled per primary region.
  • Optimization: The network is fine-tuned from ImageNet initialization with tied primary and secondary fully connected weights but separate final scoring models.The reported configuration uses learning rate 0.0001, batch size 30, two images per batch, N = 10, and 10K iterations.

4. Results

R*CNN is evaluated for action recognition from static images on three datasets: PASCAL VOC Actions, MPII Human Pose, and Stanford 40 Actions.

  • Datasets: R*CNN is evaluated on PASCAL VOC Actions, MPII Human Pose, and Stanford 40 Actions for action recognition from static images.The experiments assess the method across three action-recognition datasets.
  • Task: The evaluation targets action recognition from static images rather than video sequences.The stated experimental setting is action recognition from static images.
  • Datasets: The reported datasets span PASCAL VOC Actions, MPII Human Pose, and Stanford 40 Actions.These datasets form the paper’s named experimental evaluation set for this task.

4.1. PASCAL VOC Action

On PASCAL VOC Action, R*CNN is compared with controlled variants and published approaches while visualizations examine the contextual regions it selects. The method performs strongly across categories and can use regions capturing varied contextual cues.

  • Dataset: The PASCAL VOC Action dataset contains 10 actions plus an Other category, with person ground-truth boxes available during training and testing.Average precision is computed from estimated probabilities for all actions at test time.
  • Control Experiments: The control experiments compare Fast R-CNN using only the primary region with random-secondary and whole-scene variants.These variants isolate the effects of selecting contextual regions rather than using random regions or the entire scene.
  • Control Experiments: The Random- and Scene-RCNN settings show the value of selecting the most informative secondary region.The comparison contrasts learned selection with random selection and forced scene context.
  • Control Experiments: R*CNN performs better across all PASCAL VOC Action categories than the evaluated variants.Phoning, Reading, and Taking Photo show particularly significant gains, while Riding Bike, Riding Horse, and Running improve least.
  • Control Experiments: Using two secondary regions performs the same as one-region R*CNN with the optimal overlap parameters l = 0.2 and u = 0.75.The multi-region extension was evaluated by selecting secondary regions greedily.
  • Comparison with published results: R*CNN with l = 0.2 and u = 0.75 outperforms all other compared approaches on the PASCAL VOC Action test set.The reported advantage is especially associated with actions involving small objects or action-specific pose appearance, including Phoning, Reading, Taking Photo, and Walking.
  • Visualization of secondary regions: For Running and Walking, selected secondary regions may capture roads, body parts, or groups of people performing the action.The visualizations show that contextual cues vary with the instance.
  • Visualization of secondary regions: Top-error visualizations show red misclassified people and green selected secondary regions, with confusions linked to similar poses, objects, and related actions.Running and Walking are often confused with each other and with standing people, while object presence contributes to Taking Photo and Playing Instrument errors.

4.2. MPII Human Pose Dataset

On MPII, R∗CNN improves over RCNN for instance- and frame-level action recognition, while outperforming the best published test-set approach without motion features. Performance remains weak for several motion-dependent categories.

  • 21.7% mean AP versus 16.5% for RCNN on the MPII validation set across all actions.
  • 23% mean AP versus 18.2% for RCNN under frame-level validation evaluation.Frame-level predictions assign each action the maximum score across instances in a frame.
  • 26.7% mAP for R∗CNN versus 5.5% for the best published frame-level approach on the MPII test set.The competing approach combines Dense Trajectories with pose-specific features, whereas R∗CNN uses no motion.
  • Figure 5 relates action AP and mean AP across training-size ranges, comparing RCNN with R∗CNN on MPII validation.
  • R∗CNN performs badly on Yoga, Cooking or food preparation, and Video exercise workout, each grouped at 1.1% mean AP despite abundant examples.The authors suggest combining image and motion features to improve such categories.

4.3. Stanford 40 Actions Dataset

On Stanford 40 Actions, R∗CNN achieves high average action-recognition performance, with results varying substantially across categories.

  • 90.9% average AP on the Stanford 40 Actions test set.The dataset contains 9,532 images spanning 40 actions and is split evenly into training and test sets.
  • Performance ranges from 70.5% AP for texting message to 100% for playing violin.
  • Figure 8 reports AP separately for each action on the test set.

4.4. Attribute Classification

R∗CNN also applies to fine-grained attribute classification, using independently modeled attributes and contextual secondary regions.

  • R∗CNN achieves state-of-the-art performance on the Berkeley Attributes of People dataset.
  • Attribute prediction uses cross entropy over independent logistics because the task is multi-label.
  • Secondary regions focus on attribute-specific body parts, such as arms and torso for long sleeves and the face for hats.

Conclusion

R∗CNN extends RCNN with multiple regions and jointly learned features and models to exploit contextual cues for recognition. It outperforms published approaches on two datasets and also works for attribute classification.

  • R∗CNN adapts RCNN to use more than one region when making an action-recognition prediction.The approach uses contextual cues in addition to the person region.
  • Jointly learned features and models allow action-specific representations to emerge, while selected auxiliary regions capture different contextual modes across instances.
  • R∗CNN extends beyond action recognition to attribute classification, where secondary regions capture the region relevant to the attribute.
Loading 1505.01197v3…