Source-linked AI summary
Deep Learning-Based Human Pose Estimation: A Survey
Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, Mubarak Shah
TL;DR
Human pose estimation has advanced substantially with deep learning, but occlusion, insufficient training data, and depth ambiguity remain difficult. This survey systematically reviews deep learning-based 2D and 3D methods, their datasets, metrics, applications, and research directions, covering more than 250 papers. It finds major progress in 2D HPE while identifying persistent 3D and evaluation challenges.
Problem
Human pose estimation remains difficult under occlusion, insufficient training data, and depth ambiguity despite strong deep learning progress.
Method
The survey organizes and compares deep learning-based 2D and 3D HPE methods by input sources and inference settings, while reviewing datasets, metrics, applications, and future directions.
Results
Deep learning has significantly improved 2D HPE across single-person and multi-person methods, while 3D HPE remains challenging because monocular reconstruction is ill-posed.
Takeaways & Limitations
The survey provides a consolidated account of HPE research and uses performance comparisons to expose method strengths, weaknesses, trends, and future directions.
Takeaways & Limitations
Existing 3D HPE metrics do not assess temporal consistency and smoothness across continuous frames, leaving frame-level evaluation an open problem.
Abstract
from arXiv · showhide
Human pose estimation aims to locate the human body parts and build human body representation (e.g., body skeleton) from input data such as images and videos. It has drawn increasing attention during the past decade and has been utilized in a wide range of applications including human-computer interaction, motion analysis, augmented reality, and virtual reality. Although the recently developed deep learning-based solutions have achieved high performance in human pose estimation, there still remain challenges due to insufficient training data, depth ambiguities, and occlusion. The goal of this survey paper is to provide a comprehensive review of recent deep learning-based solutions for both 2D and 3D pose estimation via a systematic analysis and comparison of these solutions based on their input data and inference procedures. More than 250 research papers since 2014 are covered in this survey. Furthermore, 2D and 3D human pose estimation datasets and evaluation metrics are included. Quantitative performance comparisons of the reviewed methods on popular datasets are summarized and discussed. Finally, the challenges involved, applications, and future research directions are concluded. A regularly updated project page is provided: \url{https://github.com/zczcwh/DL-HPE}
1 INTRODUCTION
This survey organizes recent deep learning-based 2D and 3D human pose estimation research, addressing rapid progress alongside challenges such as occlusion, insufficient training data, and depth ambiguity. It systematically reviews methods, datasets, metrics, applications, and future directions.
- Human pose estimation estimates human body-part configurations from sensor data, especially images and videos, supporting applications including interaction, motion analysis, AR, VR, and healthcare.
- The survey tracks recent advances in deep learning-based 2D and 3D HPE to provide a clear picture of current research.
- It addresses shortcomings of earlier surveys through a systematic review of recent deep learning-based 2D and 3D methods, performance evaluations, applications, and broader discussion.
- The survey taxonomy divides 2D HPE into single-person regression or heatmap methods and multi-person top-down or bottom-up methods.
- It classifies 3D HPE by input source, distinguishing monocular RGB imagery from other sensors and further separating monocular methods by view and person count.
- The paper also summarizes datasets and evaluation metrics, compares promising methods, surveys applications such as AR/VR, and discusses future research directions.
2 2D HUMAN POSE ESTIMATION
Deep learning-based 2D human pose estimation methods are organized by person count and inference strategy, with regression and heatmap-based pipelines for single-person settings and top-down and bottom-up pipelines for multi-person settings. These methods have substantially improved performance, but occlusion, computation efficiency, and rare-pose data remain challenges.
- 2D HPE estimates human body keypoint positions from images or videos, and deep learning approaches have significantly improved results.
- 2.1 2D single-person pose estimation: Single-person methods use either regression to predict joint coordinates directly or heatmap-based prediction of joint-location probabilities.Heatmap targets are generated with 2D Gaussian kernels centered at ground-truth joint locations.
- 2.1.1 Regression methods: Regression research improves feature and pose representations through cascaded networks, structure-aware prediction, differentiable coordinate conversion, and multi-task learning.
- 2.1.2 Heatmap-based methods: Heatmap-based research develops multi-stage, encoder-decoder, multi-scale, adversarial, and body-structure-aware networks to refine joint predictions and handle difficult poses.
- 2.2 2D multi-person pose estimation: Multi-person methods are divided into top-down pipelines that detect people before estimating each pose and bottom-up pipelines that detect keypoints before grouping them into individual poses.
3 3D HUMAN POSE ESTIMATION
3D human pose estimation predicts body-joint locations in 3D space, but monocular reconstruction remains ill-posed because projection loses depth and occlusion, data scarcity, and limited generalization persist. The survey organizes deep learning approaches by input setting and inference procedure, covering skeleton estimation, mesh recovery, multi-person pipelines, alternative sensors, and key trade-offs.
- 3D HPE Challenges: Monocular 3D HPE is ill-posed because different 3D poses can produce similar 2D projections, with self-occlusion, object occlusion, depth ambiguity, and insufficient training data further complicating reconstruction.Multiple views or additional sensors can make the problem better posed through information fusion.
- Single-view Single-person 3D HPE: Single-person methods divide into skeleton-only approaches, which estimate 3D joints, and human mesh recovery methods, which use body models to reconstruct 3D meshes.Skeleton-only methods include direct estimation and 2D-to-3D lifting.
- Single-view Single-person 3D HPE: 2D-to-3D lifting first predicts 2D poses and then estimates 3D poses, generally outperforming direct estimation but remaining vulnerable to reconstruction ambiguity from over-reliance on 2D detectors.Alternative intermediate representations include 2D heatmaps and depth rankings.
- Single-view Single-person 3D HPE: Human mesh recovery incorporates parametric body models such as SMPL, while extended models address limitations including computational complexity and missing hand and facial landmarks.SMPL-X adds articulated hands and facial landmarks; PROX adds 3D environmental constraints, and SPIN combines regression with optimization.
- Single-view Single-person 3D HPE: Transformer-based HMR methods reduce resource demands: FeatER and POTTER outperform METRO using less than 10% of its parameters and 15% of its MACs.METRO and MeshGraphormer combine CNNs with transformers to regress SMPL mesh vertices from a single image.
- Single-view Multi-person 3D HPE: Top-down multi-person methods detect and process each person separately, whereas bottom-up methods estimate joints and depth maps before associating body parts using root and relative depth.Top-down methods can become computationally expensive as crowd size increases, while bottom-up methods have linear computation and time complexity but make mesh recovery less straightforward.
- Challenges and Other Sources: Major remaining challenges include poor in-the-wild generalization, severe performance drops under crowd occlusion, and the greater computational demand of 3D HPE compared with 2D HPE.Synthetic data may help diversify poses and scenes, but the synthetic-to-real distribution gap can limit its effectiveness.
4 DATASETS AND EVALUATION METRICS
The survey reviews datasets, metrics, and comparative results for deep learning-based 2D and 3D human pose estimation. It highlights dataset limitations, method trade-offs, and unresolved evaluation gaps.
- 2D Evaluation Metrics: 2D HPE evaluation uses multiple metrics because performance depends on factors such as body coverage, number of people, and human-body size.The survey summarizes PCP, PCK, AP, and AR, including threshold-based variants such as PCKh@0.5 and AP at different OKS values.
- 2D Method Comparisons: Heatmap-based 2D methods generally outperform regression approaches, but higher-resolution heatmaps increase computational cost and memory usage.Heatmaps preserve spatial information and provide probabilistic pixel-level predictions, whereas regression methods can offer fast end-to-end learning and sub-pixel accuracy.
- 3D Datasets and Results: 3D HPE datasets require difficult MoCap or wearable-IMU annotation, and methods evaluated on Human3.6M degrade quickly in more complex in-the-wild settings.Human3.6M contains 11 actors performing 17 activities in laboratory environments, limiting its coverage of real-world variation.
- 3D Evaluation Metrics: Existing 3D metrics do not jointly capture joint precision, scale robustness, temporal consistency, and pose smoothness.The survey identifies frame-level evaluation metrics for temporal consistency and smoothness as an open problem.
5 APPLICATIONS
Human pose estimation supports applications that use geometric, motion, or body-region information across recognition, interaction, content creation, and healthcare. The reviewed applications include pose-based analysis, coaching, clothing visualization, animation, AR/VR, and clinical assessment.
- Action Analysis and Surveillance: Pose information serves as a cue for action recognition, prediction, detection, tracking, anomaly detection, and privacy-preserving video surveillance.Pose-based surveillance can represent activities through poses and human meshes instead of sensitive identities.
- Action Correction and Coaching: 3D HPE and action detection support online coaching for activities requiring precise body control, including dancing, sports, and professional training.Camera-based AI coaching can provide pose correction without requiring a trainer to be physically present.
- Clothes Parsing: HPE provides plausible human body regions for clothes parsing and supports inferring a person’s 3D appearance in selected clothing.These capabilities address e-commerce needs for visualizing how clothing appears when worn.
- Animation, Movies, and Gaming: HPE can provide realistic pose information for animation, movies, and gaming while reducing reliance on expensive and complicated motion-capture equipment.Motion capture is used to represent complex movements and realistic physical interactions in these industries.
- AR and VR: AR and VR systems use human pose information to support interactive and immersive applications, including replacing people with generated cartoon characters in real-world scenes.The reviewed applications also include converting broadcast sports videos into interactive and controllable video sprites.
- Healthcare: Healthcare applications use quantitative human-motion information for disease assessment, rehabilitation training, physical therapy, and abnormal-action detection.Examples include assessing Parkinson’s disease motor severity and evaluating patients remotely at home.
6 CONCLUSION AND FUTURE DIRECTIONS
The survey synthesizes recent deep learning-based 2D and 3D HPE methods, compares their performance, and identifies unresolved challenges and future research directions. It highlights domain adaptation and architecture specialization as important open directions for improving generalization and efficiency.
- 6 CONCLUSION AND FUTURE DIRECTIONS: The survey provides a systematic overview, taxonomy, and performance comparison of recent deep learning-based 2D and 3D HPE methods.
- 6 CONCLUSION AND FUTURE DIRECTIONS: Domain adaptation remains unresolved because specialized applications may lack ground-truth data and differ in distribution from standard pose datasets.The survey notes infant images and artwork collections as examples and identifies GAN-based learning as a recent strategy for reducing domain gaps.
- 6 CONCLUSION AND FUTURE DIRECTIONS: Human body models such as SMPL, SMPL-X, GHUM, GHUML, and Adam require research on reducing their large parameter counts.
- 6 CONCLUSION AND FUTURE DIRECTIONS: Because body parts differ in movement patterns, shapes, and degrees of freedom, a single shared architecture may not optimally estimate every part.
- 6 CONCLUSION AND FUTURE DIRECTIONS: Neural Architecture Search can specialize architectures for individual body parts and discover efficient HPE networks that reduce computational cost.The survey also points to multi-objective NAS as a direction for further exploration.