PaperlayerView exact revision on arXiv

Source-linked AI summary

2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning

Diogo C. Luvizon, David Picard, Hedi Tabia

arXiv:1802.09232v2cs.CV

TL;DR

Pose estimation and action recognition are usually treated separately, despite pose’s relevance to recognizing actions. This paper proposes a jointly trained multitask architecture for 2D and 3D pose estimation and action recognition, achieving state-of-the-art results across the targeted tasks while showing benefits from end-to-end optimization.

  • Problem

    Pose estimation and action recognition are generally handled as distinct problems, while existing joint approaches do not solve both together to benefit action recognition.

  • Method

    A multitask deep architecture jointly predicts 2D and 3D poses from RGB inputs and recognizes actions using temporal joint information and visual body-part features.

  • Results

    The model reaches state-of-the-art performance on 2D and 3D pose estimation and action recognition; using only RGB frames improves NTU performance over [3] by 9.9%.

  • Takeaways & Limitations

    One efficiently shared model can address four tasks—2D and 3D pose estimation plus 2D and 3D action recognition—without requiring skeletal inputs.

Abstract

from arXiv · show

Action recognition and human pose estimation are closely related but both problems are generally handled as distinct tasks in the literature. In this work, we propose a multitask framework for jointly 2D and 3D pose estimation from still images and human action recognition from video sequences. We show that a single architecture can be used to solve the two problems in an efficient way and still achieves state-of-the-art results. Additionally, we demonstrate that optimization from end-to-end leads to significantly higher accuracy than separated learning. The proposed architecture can be trained with data from different categories simultaneously in a seamlessly way. The reported results on four datasets (MPII, Human3.6M, Penn Action and NTU) demonstrate the effectiveness of our method on the targeted tasks.

1. Introduction

The paper addresses the gap between separately handled pose estimation and action recognition by proposing a jointly trainable multitask framework. It uses differentiable pose regression to connect 2D/3D pose estimation with action recognition in one architecture.

  • Pose estimation and action recognition are challenging related tasks usually treated separately, despite pose being highly relevant to recognizing actions.
  • The unified approach handles pose estimation from single images or frame sequences and uses pose and visual information to predict actions.
  • Heat-map-based pose methods use a non-differentiable argmax post-processing step, breaking the backpropagation chain needed for joint end-to-end optimization.
  • The proposed framework extends differentiable Soft-argmax to joint 2D and 3D pose estimation, allowing action recognition to be stacked on top and trained end-to-end.
  • The pose estimator uses still images and can combine multiple dataset types, supporting 2D and 3D predictions while supplying visual features useful for action recognition.

2. Related work

Prior work commonly separates pose estimation and action recognition, with action methods often relying on localized visual features, skeleton data, or computationally expensive temporal models. This paper integrates pose prediction with recognition from RGB frames and reports state-of-the-art 3D action-recognition performance despite using temporal convolutions.

  • Human pose estimation: Pose estimation research distinguishes detection-based heat-map methods from regression-based methods that directly map inputs to joint coordinates.
  • Human pose estimation: Detection-based pose estimation typically requires argmax post-processing to recover joint coordinates, whereas regression methods output coordinates directly.
  • Action recognition: Video action recognition is difficult because it requires high-level abstraction and temporal modeling; 3D convolutions also require many parameters and substantial training memory.
  • Action recognition: Most 2D action-recognition methods use joints only to localize visual features, while methods directly using joints are limited to datasets with skeletal annotations.
  • Action recognition: The proposed model estimates poses jointly with action recognition from RGB frames, removing reliance on provided skeletons and using estimated joints to guide visual recognition.
  • 3D action recognition: The architecture predicts high-precision 3D skeletons from RGB frames and reaches state-of-the-art 3D action-recognition performance using temporal convolution instead of the more common LSTM.

3. Human pose estimation

The paper presents a differentiable regression approach for unified 2D and 3D human pose estimation, designed as the first stage of an end-to-end multitask framework. It extends Soft-argmax to 3D volumetric representations and supports joint training with mixed 2D and 3D data.

  • 3.1. Regression-based approach: The pose estimator is a differentiable regression function that maps an RGB image to the coordinates of NJ body joints.Its trainable parameters are optimized against ground-truth poses using a deep CNN.
  • 3.1.2. The Soft-argmax layer: Soft-argmax replaces the non-differentiable argmax step, converting heat maps into differentiable (x, y) joint coordinates for end-to-end optimization.The input heat map is normalized with Softmax, and the expected position approximates the maximum of the signal.
  • 3.1.1. Network architecture: The network uses an Inception-V4-based entry flow followed by K prediction blocks that progressively refine pose estimates.Intermediate visual features and joint probability maps are also available for the action-recognition stage.
  • 3.1.2. The Soft-argmax layer: A separate Sigmoid computation estimates joint visibility from the maximum value of each input heat map.Visibility is distinct from the joint probability map, which always sums to one.
  • 3.2. Unified 2D/3D pose estimation: For 3D pose, 2D heat maps are expanded into Nd volumetric heat maps, with (x, y) obtained from averaged maps and z from a one-dimensional Soft-argmax.Splitting prediction into (x, y) and z preserves 2D heat maps as a byproduct for extracting appearance features.
  • 3.2. Unified 2D/3D pose estimation: The unified approach trains on mixed 2D and 3D data by backpropagating only the (x, y) gradients for 2D examples.This combines precise 3D motion-capture data with challenging manually annotated outdoor still images.

4. Human action recognition

The framework combines pose-based and appearance-based action recognition in a multitask architecture that shares computation and integrates complementary information from body joints and visual features.

  • The action recognition framework has pose-based and appearance-based branches whose predictions are combined into the final action label.The pose branch uses body-joint coordinates, while the appearance branch uses local visual features extracted with pose probability maps.
  • 4.1. Pose-based recognition: Temporal body-joint sequences are encoded as image-like tensors, with time on the vertical axis, joints on the horizontal axis, and coordinates as channels.This representation enables classical 2D convolutions to extract patterns directly from temporal joint sequences.
  • 4.1. Pose-based recognition: A fully convolutional network produces action heat maps from input poses, using hierarchical filters to learn patterns involving relevant joints without manually selecting them.The architecture is motivated by the sparse joint dependence of actions and produces maps whose depth equals the number of actions.
  • 4.2. Appearance-based recognition: Appearance features are localized by multiplying visual feature channels with body-part probability maps, collapsing spatial dimensions, and stacking the resulting features across frames.The resulting video representation has dimensions T × NJ × Nf and is processed by an action-recognition network analogous to the pose branch.
  • 4.2. Appearance-based recognition: The multitask framework shares most computations and trains visual features across action sequences and still images, which the authors argue improves efficiency and robustness.Pose and appearance information also provide complementary cues: objects can distinguish visually similar poses, while body movements can distinguish actions not directly tied to visual objects.

5. Experiments

Experiments evaluate the multitask approach across 2D and 3D pose estimation and action recognition on four datasets. Results report strong pose and action recognition performance, with ablations showing benefits from estimated poses, full optimization, multiple clips, and feature aggregation.

  • Experimental setup: The evaluation covers 2D pose on MPII, 3D pose on Human3.6M, 2D action recognition on Penn Action, and 3D action recognition on NTU.These datasets cover four targeted task categories across 2D and 3D settings.
  • 2D pose estimation: The Soft-argmax regression method achieves results close to state of the art on MPII and is the most accurate fully differentiable approach.Performance is assessed using PCKh and AUC.
  • 3D pose estimation: The proposed method outperforms the state of the art on Human3.6M using MPJPE evaluation and generalizes 3D pose predictions from data with only 2D annotations.Training balances MPII and Human3.6M data, while evaluation follows the standard subject split.
  • 2D action recognition: The method reaches state-of-the-art Penn Action classification using RGB and estimated poses, and also improves results when using manually annotated joints.The pose-estimation training data combines MPII and Penn Action, while action recognition uses 16-frame clips.
  • 3D action recognition: 9.9% improvement over [3] is reported on NTU using only RGB frames, while RGB frames and 3D predicted poses improve the state of the art overall.NTU training uses mixed data and 20-frame clips; temporal information is modeled with convolution rather than LSTM.
  • Ablation study: 2.9% accuracy gain comes from estimated poses over Kinect poses, 3.3% from full optimization, and 1.1% from averaging multiple video clips.On Penn Action, joint learning reaches 97.3%, only 0.1% below sequential learning followed by fine tuning.

6. Conclusions

The paper presents one heavily shared multitask architecture that jointly estimates 2D and 3D poses and recognizes actions from RGB videos. Experiments show it can match or outperform dedicated approaches across all four tasks.

  • Conclusions: The model predicts 2D and 3D body-joint locations from RGB frames before using temporal joint coordinates and body-part appearance to predict actions.Action prediction combines semantic information from pose evolution with visual information from attention-based pooling.
  • Conclusions: A single efficiently shared model addresses 2D pose estimation, 3D pose estimation, 2D action recognition, and 3D action recognition.The conclusion attributes this to heavy sharing of weights and features.
  • Conclusions: The reported experiments show performance equal to or better than dedicated approaches on all four tasks.The conclusion summarizes the paper-wide outcome rather than a single benchmark result.

Appendix A: Network architecture

The network architecture uses a shared Inception-V4-based stem with separable residual modules, followed by task-specific pose and action prediction components.

  • Shared architecture: The separable residual module supports differing or equal input and output feature sizes while operating on W × H feature maps with S × S filters.The filters are usually 3 × 3 or 5 × 5, and C denotes simple 2D convolution.
  • Shared architecture: The shared network entry flow is based on Inception-V4 and uses convolutional and separable residual components.The separable residual module is built from depth-wise separable convolutions.
  • Pose estimation: The pose prediction block estimates pose outputs using Nd depth heat maps per joint and NJ body joints.The block combines convolutional and separable residual modules.

Appendix B: Training parameters

Training merges heterogeneous pose datasets through a common joint layout and uses separate optimization procedures for pose regression and action recognition. The implementation also specifies multi-clip architecture and extensive spatial-temporal augmentation settings.

  • Data integration: Merged datasets are converted to a common pose layout with a fixed joint count, while unavailable joints receive invalid values excluded from the loss.The implementation also alternates human pose layouts to resemble the Penn Action layout.
  • Optimization: Pose regression uses RMSprop with an initial learning rate of 0.001 and batch size 24, while action recognition uses SGD with Nesterov momentum of 0.98.Both learning rates are reduced by a factor of 0.2 when validation performance plateaus.
  • Evaluation settings: Human3.6M pose evaluation compares single-dataset training with mixed Human3.6M and MPII data using single-crop and multi-crop settings.Table 6 reports averaged joint error on reconstructed poses.
  • Architecture settings: Action recognition uses four prediction blocks, whereas pose estimation uses eight, with 256 × 256 cropped RGB images across experiments.The architecture specifies K = 4 for action recognition and K = 8 for pose estimation.

Appendix C: Additional experiments

This appendix examines how using multiple datasets during training contributes to 3D pose estimation. Table 6 compares training with Human3.6M alone against training with Human3.6M and MPII together.

  • Appendix C: Additional experiments: Table 6 reports additional 3D pose-estimation results for models trained with different dataset combinations.The comparison is intended to show the contribution of multiple datasets during training.
  • Appendix C: Additional experiments: One training condition uses Human3.6M only.
  • Appendix C: Additional experiments: The other training condition combines Human3.6M and MPII datasets.
Loading 1802.09232v2…