Source-linked AI summary
Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
Diogo C Luvizon, Hedi Tabia, David Picard
TL;DR
Human pose estimation and action recognition are usually handled separately despite their shared dependence on body representation. This paper introduces an end-to-end multi-task architecture that jointly predicts poses and actions from RGB images and video, achieving strong performance across both tasks while operating above 100 FPS.
Problem
Pose estimation and action recognition are related but are usually treated as separate tasks, limiting joint end-to-end optimization.
Method
The paper uses a shared multi-task architecture that jointly estimates 2D/3D poses and recognizes actions from still images and video clips.
Results
The method achieves state-of-the-art or comparable results across pose estimation and action recognition while running at more than 100 frames per second.
Takeaways & Limitations
Joint learning improves action recognition and provides a scalable model whose inference depth can be selected after training for different speed–accuracy trade-offs.
Takeaways & Limitations
Decoupled pose predictions can drift toward image regions more informative for action recognition, such as objects held in the hands.
Abstract
from arXiv · showhide
Human pose estimation and action recognition are related tasks since both problems are strongly dependent on the human body representation and analysis. Nonetheless, most recent methods in the literature handle the two problems separately. In this work, we propose a multi-task framework for jointly estimating 2D or 3D human poses from monocular color images and classifying human actions from video sequences. We show that a single architecture can be used to solve both problems in an efficient way and still achieves state-of-the-art or comparable results at each task while running at more than 100 frames per second. The proposed method benefits from high parameters sharing between the two tasks by unifying still images and video clips processing in a single pipeline, allowing the model to be trained with data from different categories simultaneously and in a seamlessly way. Additionally, we provide important insights for end-to-end training the proposed multi-task model by decoupling key prediction parts, which consistently leads to better accuracy on both tasks. The reported results on four datasets (MPII, Human3.6M, Penn Action and NTU RGB+D) demonstrate the effectiveness of our method on the targeted tasks. Our source code and trained weights are publicly available at https://github.com/dluvizon/deephar.
1. Introduction
The paper addresses the separation of human pose estimation and action recognition by proposing a jointly trainable multi-task framework. Its architecture supports parallel pose and action prediction, multimodal training, and improved efficiency through shared processing.
- The paper proposes an end-to-end multi-task framework that jointly estimates 2D/3D human poses and recognizes actions.
- The method extends differentiable soft-argmax to enable joint 2D and 3D pose estimation while preserving end-to-end training for action recognition.
- Pose and action predictions are made in parallel at multiple feature-map resolutions and repeatedly refined through supervised prediction blocks.
- The framework trains with mixed still-image and video data, including 2D and 3D pose annotations, while sharing computations across tasks.
- The reported method achieves state-of-the-art performance on both 2D and 3D pose scenarios and supports scalable speed–accuracy trade-offs after training.
2. Related Work
Prior work largely treats pose estimation and action recognition as separate problems, with limitations in differentiability, temporal modeling, data use, or pose quality. The proposed approach jointly estimates poses from RGB frames and uses them for action recognition.
- Pose-estimation methods include heat-map detection and direct regression, but detection pipelines generally require argmax post-processing to recover joint coordinates.
- Soft-argmax provides a differentiable alternative for computing body-joint coordinates from heat maps.
- 3D pose methods using volumetric heat maps incur substantial parameter and memory costs, motivating more efficient representations.
- Earlier literature commonly separates pose estimation from action recognition, while the proposed approach performs both jointly from RGB input.
- The model predicts 3D poses from RGB frames instead of relying on noisy Kinect skeletons and reaches state-of-the-art 3D action-recognition performance.
3. Proposed Multi-task Approach
The proposed method jointly processes still RGB images and video clips to estimate poses and recognize actions. It outputs per-frame pose and confidence predictions, plus action probabilities for video inputs.
- The method accepts either a still RGB image or a video clip, treating pose estimation as a single-frame task.
- For each frame, it predicts a human pose and per-joint confidence scores, while video clips additionally produce action probabilities.
3.1. Network Architecture
The network uses shared entry features, multi-scale downscaling and upscaling pyramids, and prediction blocks that supervise and refine pose and action outputs. Separate information streams support single-frame pose processing and video action recognition.
- The architecture extracts entry-flow features and processes them through repeated downscaling and upscaling pyramids containing prediction blocks and skip connections.
- Each prediction block produces supervised pose and action predictions, reinjecting them into later layers for further refinement.
- Single-frame mode activates pose-estimation layers, whereas video-clip mode activates both pose and action-recognition layers.
- Pose and action information are independently propagated between prediction blocks through their respective feature streams.
- The network combines shared multi-task features with single-frame pose features and video-only action features, all represented as tensors compatible with 2D convolutions.
- Pose outputs include joint probability maps and depth maps, with depth normalized relative to the root joint in [0, 1].
3.2. Pose Regression
The method regresses 2D coordinates with a differentiable soft-argmax and estimates 3D pose by combining 2D locations with depth maps. Prediction re-injection enables iterative pose refinement across resolutions.
- 3.2. Pose Regression: 2D joint coordinates are obtained as expected positions from normalized probability maps, keeping the regression differentiable and resolution-invariant.The coordinates are constrained to normalized image limits.
- 3.2. Pose Regression: 3D pose concatenates 2D coordinates with depth coordinates estimated by spatially pooling each joint’s depth map using its probability map.This replaces volumetric heat maps with specialized depth maps.
- 3.2. Pose Regression: Joint confidence is computed from the maximum probability-map response, decreasing when the map is diffuse or multimodal.A peaked map produces confidence close to one.
- 3.2. Pose Regression: Pose predictions are re-injected into later prediction blocks, allowing refinement from features at different pyramid levels and resolutions.The re-injected information includes 2D pose and depth predictions.
3.3. Human Action Recognition
Action recognition combines pose-derived temporal features with localized appearance features within the shared multi-task architecture. Appearance processing is computationally efficient, while action-feature re-injection improves recognition without additional parameters; decoupled poses address competing gradients.
- 3.3. Human Action Recognition: Pose sequences are encoded as image-like tensors, with time as rows, joints as columns, and coordinates as channels for 2D or 3D action features.This representation permits classical 2D convolutions over temporal joint patterns.
- 3.3. Human Action Recognition: Localized appearance features complement pose features by encoding contextual visual information around body joints.Probability maps localize multitask feature channels before spatial aggregation into video-clip appearance features.
- 3.3. Human Action Recognition: Shared computation makes appearance processing efficient, while simultaneous training on related tasks and datasets is argued to produce more robust visual features.The framework uses both action sequences and still images captured in the wild.
- 3.3. Human Action Recognition: Action-feature re-injection improves action recognition without adding parameters by refining predictions across successive processing blocks.The mechanism parallels re-injection used for single-frame pose features.
- 3.3. Human Action Recognition: Decoupled action poses absorb action gradients while original pose predictions remain specialized for pose estimation.A replicated final projection layer is introduced late in training and remains coherent with supervised poses.
4. Experiments
Experiments evaluate the joint framework across 2D and 3D pose estimation and action recognition using four public datasets. The protocol reports task-specific pose and classification metrics under standard dataset splits and clip settings.
- 4. Experiments: The experiments cover 2D pose estimation, 3D pose estimation, and action recognition across MPII, Human3.6M, Penn Action, and NTU RGB+D.The scenarios pair MPII with Penn Action for 2D tasks and include Human3.6M and NTU for 3D settings.
- 4. Experiments: Action recognition is evaluated with percentage of correct action classification using a 50/50 Penn Action split and NTU’s cross-subject protocol.NTU uses 20 subjects for training and the remaining subjects for testing.
- 4. Experiments: Action predictions are evaluated from either one centered clip or multiple temporally spaced clips whose decisions are averaged per video.The multi-clip spacing is T/2 frames.
4.2. Implementation and Training Details
Training uses an elastic-net pose loss for both 2D and 3D coordinates, lightweight separable convolutions for single-frame processing, and staged initialization before joint optimization. The architecture uses four levels and up to eight pyramids.
- 4.2. Implementation and Training Details: The same elastic-net loss trains 2D and 3D pose predictions, using only coordinates available in each dataset.Estimated and ground-truth joint positions are compared for each body joint.
- 4.2. Implementation and Training Details: Separable 5×5 convolutions are used for single-frame layers, standard 3×3 convolutions for action layers, and the architecture uses four levels with up to eight pyramids.More than eight pyramids produced no further significant pose-estimation improvement.
- 4.2. Implementation and Training Details: Training proceeds from pose-only initialization to action-only training and finally simultaneous multi-task optimization.Pose pretraining provides general initialization, while intermediate action training initializes the action branch on top of the pose estimator.
- 4.2. Implementation and Training Details: Optimization uses RMSprop for pose initialization and Penn Action, and Nesterov SGD for NTU, with scheduled learning-rate reductions.Joint-confidence and action losses are down-weighted by a factor of 0.01.
4.3. Evaluation on 3D Pose Estimation
The multi-task model achieves state-of-the-art 3D pose estimation on Human3.6M, while a single model and training procedure produces the reported pose and action results.
- 48.6 millimeters is the multi-task model’s average MPJPE on Human3.6M, improving the previous work by 4.6 mm.
- 49.5 mm using pose-only training versus 48.6 mm with multi-task training indicates a 0.9 mm improvement from joint training.
- A single model and training procedure produced all reported 3D pose estimation and 3D action recognition images and scores.
4.4. Evaluation on Action Recognition
The framework achieves strong action-recognition performance in both 2D and 3D settings using shared multi-task models trained with pose and action data.
- A single model combines MPII pose estimation with Penn Action clips for 2D recognition and Human3.6M, MPII, and NTU data for 3D recognition.
- 98.7% is achieved on Penn Action, outperforming previous methods, including methods using manually annotated poses.
- 89.9% of actions are correctly classified on NTU’s 60-action cross-subject split, improving previous results by at least 3.3%.
4.5. Ablation Study
Ablations show that supervision, complementary pose and appearance features, and multi-task training improve performance, while model truncation enables faster inference with some degradation.
- 4.5.1 Network Design: Supervising all prediction blocks is distinguished from supervising only the final block, while Table 4 evaluates these layouts on MPII and Penn Action.
- 4.5.1 Network Design: 88.3% on MPII and 98.2% on Penn Action are reached by the full network with one multi-task model.
- 4.5.2 Pose and Appearance Features: Combined pose and appearance features reach higher Penn Action accuracy than pose-only or appearance-only features, which score 97.4% and 97.9%.
- 4.5.2 Pose and Appearance Features: Decoupled action poses usually remain coherent with supervised poses but can drift toward image regions more informative for recognizing actions.
- 4.5.3 Single-task vs. multi-task: Adding pose supervision improves Penn Action recognition by almost 10%, while multi-tasking improves NTU recognition by 1.9%.
- 4.5.4 Inference Speed: More than 90 FPS supports the best predictions; at over 180 FPS, 3D pose reaches 57.3 millimeters error, while NTU action recognition reaches 87.7%.
5. Conclusion
The proposed multi-task approach jointly handles human pose estimation and action recognition through efficient supervision and mixed-data training. It improves both tasks relative to separate learning while supporting seamless training across still images and video clips.
- 5. Conclusion: Multi-scale pose and action supervision with re-injection yields an efficient densely supervised architecture.The model refines predictions across feature-map resolutions during training.
- 5. Conclusion: Mixed 2D and 3D training combines precise indoor 3D data with in-the-wild images annotated with 2D poses.The authors report significant improvements for 3D pose estimation from this training strategy.
- 5. Conclusion: The evaluation covers pose estimation and action classification on MPII, Human3.6M, Penn Action, and NTU RGB+D.Table 7 reports PCKh, MPJPE, and action-classification accuracy across these tasks and datasets.
- 5. Conclusion: The model trains single frames and video clips simultaneously and can be cut at different levels for pose or action prediction.This supports a seamless and scalable deployment approach.
- 5. Conclusion: Joint learning produces a better solution for pose estimation and action recognition than learning the tasks separately.The authors also report consistent action-recognition improvement from jointly learning human poses.