PaperlayerView exact revision on arXiv

Source-linked AI summary

Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods

Yucheng Chen, Yingli Tian, Mingyi He

arXiv:2006.01423v1cs.CV

TL;DR

Monocular human pose estimation seeks human posture from images or video but remains challenged by occlusion, crowded people, and complex scenes. This survey reviews deep learning-based 2D and 3D methods published mainly since 2014, covering frameworks, datasets, metrics, comparisons, and future directions. It identifies distinct method trade-offs, including top-down accuracy versus speed and heatmap representations’ suitability for training.

  • Problem

    Monocular HPE must estimate human posture despite flexible configurations, diverse appearances, complex environments, occlusion, and crowded people.

  • Method

    The survey synthesizes recent deep learning-based 2D and 3D HPE methods, organizing approaches by tasks and reviewing frameworks, datasets, evaluation metrics, and comparisons.

  • Results

    The survey reports that top-down methods achieve state-of-the-art performance on almost all benchmarks but are speed-limited, while bottom-up methods can run in real time but are affected by complex backgrounds and occlusions.

  • Takeaways & Limitations

    Heatmap representations are more suitable for deep learning training than coordinate representations, and efficient networks plus adequate diverse data remain important requirements.

  • Takeaways & Limitations

    Monocular HPE still has unresolved challenges and a gap between research and practical applications, especially with body-part occlusion and crowded people.

Abstract

from arXiv · show

Vision-based monocular human pose estimation, as one of the most fundamental and challenging problems in computer vision, aims to obtain posture of the human body from input images or video sequences. The recent developments of deep learning techniques have been brought significant progress and remarkable breakthroughs in the field of human pose estimation. This survey extensively reviews the recent deep learning-based 2D and 3D human pose estimation methods published since 2014. This paper summarizes the challenges, main frameworks, benchmark datasets, evaluation metrics, performance comparison, and discusses some promising future research directions.

1. Introduction

Monocular human pose estimation is an important vision task advanced by deep learning, but it remains difficult because of flexible bodies, varied appearances, and complex environments. This survey reviews recent methods and organizes their applications, categories, datasets, and research directions.

  • Motivation: Deep learning has driven rapid progress in vision-based human pose estimation through stronger networks, richer datasets, and more practical body models.The survey focuses on methods developed mainly since 2014.
  • Applications: HPE supports applications including action recognition, tracking, virtual reality, human-computer interaction, surveillance, medical assistance, self-driving, and sports analysis.Sports analysis can quantify indicators such as running distance and number of jumps.
  • Challenges: Monocular HPE faces flexible body configurations, diverse appearances, and complex environments that produce self-occlusion, unusual poses, and truncation.Nearby-person overlap and varied viewing angles further complicate estimation.
  • Survey scope: The survey differs from earlier reviews by extensively summarizing recent milestone deep learning-based HPE methods, while incorporating earlier reviewed work for comprehensive coverage.Most recent advances had not been presented in any survey before this work.

2.1. HPE Method Categories

Deep learning-based monocular HPE methods are organized along four complementary distinctions: body-model use, prediction direction, formulation, and training stages. These categories clarify major design choices and their associated trade-offs.

  • Category overview: HPE methods are categorized as generative or discriminative, top-down or bottom-up, regression-based or detection-based, and one-stage or multi-stage.The categories reflect body-model use, processing direction, problem formulation, and training structure.
  • Generative vs. discriminative: Generative methods use human body models, whereas discriminative methods learn or search mappings to pose space without body models.Discriminative methods are usually faster but may be less robust for poses absent from training.
  • Top-down vs. bottom-up: Top-down methods detect people before estimating each pose, while bottom-up methods predict body parts first and group them by person.Bottom-up processing can struggle to group corresponding parts when people overlap substantially.
  • Regression vs. detection: Regression-based methods directly predict joint coordinates or body-model parameters, whereas detection-based methods use body-part patches or joint heatmaps.Coordinate regression is highly nonlinear, while small-region representations provide denser pixel information but can limit final coordinate accuracy.
  • One-stage vs. multi-stage: One-stage methods map images to poses end-to-end, whereas multi-stage methods use sequential prediction and intermediate supervision or constraints.One-stage training is easier, but it provides fewer intermediate constraints.

2.2. Human Body models

Human body models provide task-specific representations for the flexible, complex human body. The survey distinguishes skeleton-, contour-, and volume-based models used across HPE settings.

  • Overview: Human body modeling is a key HPE component that represents attributes such as kinematic structure, body shape, surface texture, and joint positions.A model need not represent every body attribute if it satisfies the requirements of its task.
  • Skeleton-based model: Skeleton-based models represent joint locations and limb orientations according to skeletal structure, often as graphs of joints and connections.These models typically contain 10 to 30 joints.
  • Contour-based model: Contour-based models capture approximate limb width and torso contours using rectangles, boundaries, or person silhouettes.Cardboard models and Active Shape Models are cited examples.
  • Volume-based model: Volume-based models represent 3D body shapes and poses with geometric primitives or meshes, including SCAPE and SMPL.Modern volume-based models are generally mesh representations derived from 3D scans.
  • Relation to HPE: The survey applies these body-model concepts within its review of 2D and 3D human pose estimation approaches.The 2D section concerns locating human joints from monocular images or videos.

3.1. 2D single person pose estimation

2D single-person pose estimation methods are organized around regression and detection formulations, with heatmaps becoming the dominant representation because they provide denser supervision and greater robustness. Research also advances through refinement, body-structure modeling, multi-task learning, and temporal information.

  • CNN-based 2D single-person methods are classified as regression-based or detection-based according to how they formulate pose prediction.
  • Regression methods directly map images to joint coordinates, whereas detection methods predict body-part locations using patches or joint heatmaps.
  • Regression-based methods: Early DeepPose regressed joint coordinates from full images without body models or part detectors, using cascaded multi-stage regressors for refinement.
  • Regression-based methods: Later approaches combine heatmaps with differentiable coordinate transforms, including Soft-argmax and DSNT, to retain dense supervision while producing numerical joint positions.
  • Other methods improve pose prediction through iterative refinement, body-structure modeling, multi-task learning, adversarial supervision, and temporal information from video.
  • Detection-based methods: Heatmaps encode each joint as a Gaussian-centered channel, providing denser supervision and greater robustness than direct coordinate representation.Most recent research therefore uses heatmap representations, and heatmap training is summarized as more suitable than coordinate representation.

3.2. 2D multi-person pose estimation

2D multi-person pose estimation must detect and localize an unknown number of people, motivating top-down and bottom-up pipelines. Top-down methods generally achieve stronger benchmark performance but depend on person detection and scale with the number of people, while bottom-up methods are faster but vulnerable to background complexity and occlusion.

  • Multi-person pose estimation requires both person detection and joint localization because the number of people is not given in the input image.
  • Top-down methods: Top-down methods detect person bounding boxes first, then apply a single-person estimator to each detected person.
  • Top-down methods: Top-down performance depends on person-detection precision, and runtime grows with the number of detected people.
  • Bottom-up methods: Bottom-up methods detect all body parts first and group them into independent skeletons, using either separate or single-stage detection-and-grouping designs.DeepCut separates candidate detection and grouping, whereas associative embedding performs both predictions in one stage.
  • Bottom-up methods: Bottom-up processing can run in real time, but complex backgrounds and human occlusions can substantially influence performance.
  • Top-down methods: Top-down approaches achieve state-of-the-art performance on almost all benchmark datasets, whereas their processing speed is usually not real-time.

4.1. 3D single person pose estimation

3D single-person pose estimation predicts body-joint depth from monocular images, using model-free or model-based strategies. Model-free methods either regress 3D pose directly or extend intermediate 2D pose estimates, while model-based methods impose parametric or kinematic body structure.

  • Challenges: 3D HPE is harder than 2D HPE because it must predict joint depth, while available training datasets are fewer and often constrained in generalizability.Most 3D datasets are collected in constrained environments.
  • Model-free methods: Model-free methods either directly map images to 3D poses or estimate depth from an intermediate 2D pose.The latter strategy can reuse 2D pose-estimation methods and datasets.
  • Model-free methods: Direct 3D methods include shallow coordinate regression and volumetric coarse-to-fine prediction with multi-stage refinement.These approaches differ in whether they regress coordinates directly or represent pose volumetrically.
  • Model-free methods: 2D-to-3D methods use matching, Euclidean Distance Matrices, depth rankings, or dedicated predictors to infer 3D joint positions.Some methods also encode 2D poses into formats such as 3D BVH for rendering.
  • Model-based methods: Model-based methods estimate pose and shape through parametric body models, including SMPL, kinematic trees, and learned latent 3D pose models.These models typically separate body pose and shape or encode articulated structure as a prior.

4.2. 3D multi-person pose estimation

Monocular 3D multi-person pose estimation is a newer field with relatively few methods. Existing approaches build on 3D single-person estimation and use bottom-up or multi-stage detection, proposal, and refinement pipelines.

  • Overview: 3D multi-person pose estimation remains a relatively new research area with only a few proposed methods.The surveyed methods are summarized in Table 6.
  • Bottom-up methods: Bottom-up methods use 2D pose and part affinity fields to infer person instances, while occlusion-robust pose maps provide multi-person occlusion information.These components support grouping poses across people under occlusion.
  • Multi-stage methods: LCR-Net uses three stages: person detection, anchor-pose assignment by classification, and pose refinement by regression.The pipeline begins with Faster R-CNN person localization.

5. Datasets and evaluation protocols

The survey treats datasets as foundational for training and fair comparison, covering 2D and 3D resources, annotation characteristics, evaluation methods, and reported benchmark performance. It emphasizes that dataset scale, diversity, and realism shape the usefulness of deep-learning pose estimation.

  • Dataset role: Datasets support fair algorithm comparison and add challenges through greater quantity, complexity, and improved collection practices.Recent datasets are less limited by data volume or laboratory environments because of motion capture and crowdsourcing.
  • Evaluation protocols: The survey describes dataset characteristics, evaluation methods, recent state-of-the-art performance, and annotation collection procedures.These materials are presented for both 2D and 3D human pose estimation.
  • 2D datasets: Earlier 2D datasets often have few scenes, monotonous viewpoints, limited activities, and too few images for robust deep-network training.The survey therefore selects 2D datasets with more than 1,000 training images.
  • 2D datasets: FLIC and LSP provide movie- or sports-oriented 2D annotations, while MPII and COCO support broader articulated-pose evaluation in varied imagery.MPII includes diverse activities, and COCO keypoint annotations cover 17 body joints with visibility and left/right labels.
  • Video datasets: Video pose datasets extend static-image resources with dense frame annotations, enabling temporal modeling and action-recognition applications.They include datasets focused on single individuals as well as multiple people.
  • Dataset limitations: Some upper-body video datasets use simple scenes, static views, and characters with small motion ranges, limiting their realism.This constraint is explicitly noted for the described dataset group.

6. Conclusion and Future Research Directions

The survey organizes deep-learning monocular HPE into four task categories and reviews methods, datasets, and evaluation protocols. It concludes that occlusion, crowding, efficiency, and training-data limitations remain important barriers to practical deployment, motivating richer context, body-structure priors, and more diverse data.

  • Conclusion: The survey reviews 2D and 3D monocular HPE across single-person and multi-person tasks, while summarizing datasets and evaluation protocols.It organizes the literature into four task categories.
  • Open challenges: Occlusion and crowded people remain unresolved challenges, and a gap persists between research results and practical applications.The conclusion identifies efficient networks and adequate training data as important requirements.
  • Future networks: Future networks should combine global and local context with human-body structure constraints, while retaining effective designs such as multi-stage processing and multi-scale fusion.The survey also identifies network efficiency as important for real-life applications.
  • Future data: More diverse data can improve robustness to irregular poses, occlusion, and crowds, although synthetic data retain a domain gap from real data.Cross-dataset supplementation, including 2D data for 3D datasets, can address insufficient training diversity.
Loading 2006.01423v1…