PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspective

Wu Liu, Qian Bao, Yu Sun, Tao Mei

arXiv:2104.11536v1cs.CV

TL;DR

Monocular human pose estimation remains difficult because complex scenes, occlusion, and limited diverse 3D data challenge both 2D and 3D methods. This survey unifies deep learning approaches across 2D and 3D tasks, summarizes their performance and strategies, and identifies future research challenges.

  • Problem

    Monocular pose estimation must handle complex postures, occlusion, crowded scenes, and insufficient diverse in-the-wild 3D training data.

  • Method

    The paper surveys over 200 deep learning methods for monocular 2D and 3D pose estimation under unified frameworks, covering representations, benchmarks, metrics, and task-specific designs.

  • Results

    The survey reports significant progress in 2D and 3D pose estimation, while finding that 2D supervision and weak or unsupervised methods address 3D-data scarcity to some extent.

  • Takeaways & Limitations

    The synthesis shows that pose-specific architectures and richer body representations are central directions for improving accuracy and efficiency across challenging monocular settings.

  • Takeaways & Limitations

    Models trained only on restrained datasets such as Human3.6M are prone to fail on in-the-wild images because pose, shape, and scene diversity are limited.

Abstract

from arXiv · show

Estimation of the human pose from a monocular camera has been an emerging research topic in the computer vision community with many applications. Recently, benefited from the deep learning technologies, a significant amount of research efforts have greatly advanced the monocular human pose estimation both in 2D and 3D areas. Although there have been some works to summarize the different approaches, it still remains challenging for researchers to have an in-depth view of how these approaches work. In this paper, we provide a comprehensive and holistic 2D-to-3D perspective to tackle this problem. We categorize the mainstream and milestone approaches since the year 2014 under unified frameworks. By systematically summarizing the differences and connections between these approaches, we further analyze the solutions for challenging cases, such as the lack of data, the inherent ambiguity between 2D and 3D, and the complex multi-person scenarios. We also summarize the pose representation styles, benchmarks, evaluation metrics, and the quantitative performance of popular approaches. Finally, we discuss the challenges and give deep thinking of promising directions for future research. We believe this survey will provide the readers with a deep and insightful understanding of monocular human pose estimation.

I. INTRODUCTION

Monocular human pose estimation predicts 2D or 3D body information from images or video, and deep learning has substantially advanced both accuracy and efficiency. This survey organizes 2D-to-3D methods under unified frameworks while examining data scarcity, 2D–3D ambiguity, occlusion, and multi-person complexity.

  • Monocular human pose estimation predicts body-joint locations and, increasingly, body shape or texture from a single image or video.
  • Deep learning and larger datasets have advanced pose-estimation accuracy and efficiency since 2014.
  • The survey addresses a gap in prior reviews by jointly analyzing deep-learning-based 2D and 3D methods and their intrinsic connections.
  • Challenges: Monocular estimation remains difficult under complex postures, occlusion, truncation, blur, low resolution, and crowded scenes.
  • Deep Learning Framework: Most single-person networks use a pose encoder followed by a pose decoder, with designs including multi-scale fusion, multistage processing, refinement, and multitask learning.
  • Challenges: Limited in-the-wild 3D data motivates using 2D pose as an intermediate representation or extra supervision, but a single 2D pose can correspond to multiple 3D poses.
  • Scope: The survey systematically covers 2D and 3D approaches, benchmarks, state-of-the-art comparisons, and future research directions.

II. BACKGROUND

Human-body pose can be represented through keypoints or structured body models. These representations range from coordinates and heatmaps to hierarchical bones, volumetric parts, and detailed statistical meshes.

  • Existing human-body representations divide into keypoint-based and model-based categories.
  • Keypoint-Based Representation: Keypoint-based representations describe body joints using 2D or 3D coordinates connected according to body structure.
  • Keypoint-Based Representation: Heatmaps convert keypoint coordinates into spatial response maps, while orientation maps provide auxiliary information about body-part directions.
  • Keypoint-Based Representation: Hierarchical bone vectors represent 3D skeletons as parent-to-child vectors organized along a kinematic tree.
  • Model-Based Representation: Model-based representations encode richer structural information through part-based volumetric models or statistical 3D human-body models.
  • Model-Based Representation: SMPL separates body shape and pose and encodes the 3D mesh with low-dimensional parameters.

B. 3D-to-2D Projection

3D-to-2D projection connects 3D pose representations with image-plane observations through camera models. The survey describes perspective and weak-perspective formulations and situates projection within monocular pose-estimation pipelines.

  • 3D-to-2D projection uses camera models to generate pose pairs, supervise 3D poses from 2D annotations, or refine 2D poses through projected 3D poses.
  • Perspective camera model: Perspective projection transforms 3D points into camera coordinates with extrinsic parameters, then applies the intrinsic matrix for image-plane adjustment.
  • Weak-perspective camera model: Weak-perspective projection is widely used for uncalibrated images because complete perspective-camera parameters are difficult to retrieve.It models projection with global rotation, orthographic projection, image-plane translation, and scale.
  • Single-person estimation framework: Single-person pose-estimation networks commonly use a pose encoder followed by a decoder that predicts keypoint locations through regression or detection.
  • Pose decoder: Detection-based decoders generate keypoint heatmaps and are widely used because direct coordinate regression may fail for complex poses.

B. Multi-person Pose Estimation

Multi-person pose estimation must locate keypoints for an unknown number of people. Its representative processing paradigms are top-down and bottom-up methods.

  • Multi-person pose estimation detects and locates keypoints for all people in an image when the number of people is unknown in advance.
  • Top-down methods: Top-down methods first detect every person, then estimate keypoints separately within each detected bounding box.
  • Bottom-up methods: Bottom-up methods predict keypoints across the image and then group them into individual people.

1) Top-down Methods:

Top-down multi-person methods crop detected people and apply single-person estimators, while related approaches improve prediction through multi-tasking, refinement, and multi-stage fusion.

  • Top-down Methods: Top-down methods detect and crop each person before applying a single-person pose model and post-processing such as pose NMS.
  • Top-down Methods: G-RMI combines Faster R-CNN person detection with fully convolutional heatmap-and-offset prediction and keypoint-based NMS.
  • Top-down Methods: Multi-task networks jointly predict pose with related outputs such as person detection, semantic-part segmentation, or person segmentation.
  • Top-down Methods: The WholeBody dataset extends COCO with whole-body annotations.
  • Top-down Methods: CPN uses global and refining networks with online hard keypoints mining to progressively refine difficult keypoint predictions.It achieved first place in the COCO 2017 keypoint challenge.
  • Top-down Methods: Crowded, occluded, and truncated scenes motivate methods using spatial transformers, pose-guided proposals, pose NMS, and confusing-sample mining.
  • Bottom-up Methods: Bottom-up methods predict all image keypoints before grouping candidates, using association strategies such as PAFs or associative embeddings instead of costly ILP.
  • Bottom-up Methods: OpenPose jointly learns keypoint locations and limb associations with 2D Part Affinity Fields, then uses bipartite matching for person assembly.Associative Embedding instead predicts heatmaps and tags, matching detections that share embedding tags.

C. 2D Pose Estimation in Videos

Video pose estimation extends image-based methods by exploiting temporal information to reduce motion blur and geometric inconsistency. Single- and multi-person systems use propagation, sequence models, distillation, tracking, and spatio-temporal optimization.

  • Video pose estimation must model temporal relations across frames because directly applying image-based methods can produce sub-optimal results under motion blur and geometric inconsistency.
  • Single-person video estimation: Single-person video methods propagate temporal clues through frame inputs, optical flow, recurrent sequence models, or action-recognition branches.
  • Single-person video estimation: Dynamic Kernel Distillation transfers pose knowledge through a one-shot feed-forward model that predicts temporally coherent pose kernels.
  • Multi-person video estimation: Multi-person video estimation must handle occlusion, motion blur, and pose or appearance variation, often together with articulated tracking.PoseTrack provides a large-scale in-the-wild benchmark with hundreds of videos and more than 150K poses.
  • Top-down methods: Top-down video methods detect people and keypoints per frame, then propagate boxes or keypoints across frames for tracking.
  • Top-down methods: Spatio-temporal merging can smooth outputs and handle failed detections in scenes with severe occlusion and highly entangled people.
  • Bottom-up methods: Bottom-up video methods predict frame-level keypoints and associate them across frames through spatio-temporal optimization or graph partitioning.
  • Summary: The survey summarizes video methods around efficiently propagating spatio-temporal information to maintain prediction consistency and smoothness.

D. 2D Pose Estimation for Other Tasks

2D human pose information supports diverse vision tasks by providing body structure, part details, and alignment cues. The cited applications include person re-identification, action recognition, human-object interaction, and human parsing.

  • Person Re-Identification: Pose-derived body-part information helps discriminate individuals under occlusion and partial visibility in person re-identification.Pose-driven response maps can alleviate pose variations during feature extraction and matching.
  • Action Recognition: Pose detection and alignment provide discriminative information for recognizing human actions in dynamic scenes.Pose estimation can extract appearance and flow information at characteristic body positions.
  • Human-Object Interactions: Human pose supports human-object interaction recognition by identifying keypoints, body parts, and their spatial correlations.Pose estimators can predict keypoints used to construct spatial-pose information and body-part attention.
  • Human Parsing: Pose supplies structural information for segmenting and labeling fine-grained human body parts in human parsing.A unified framework can perform human parsing and pose estimation simultaneously.
  • 3D Pose Context: Monocular 3D pose estimation predicts 3D joints or body meshes and is more challenging than 2D estimation because depth is unavailable.Skeleton-based methods output joint locations, whereas mesh-based methods output 3D bodies using topology or statistical models.

A. Skeleton-based 3D Pose Estimation

Skeleton-based monocular 3D pose estimation uses volumetric heatmaps, 2D-to-3D lifting, or image-feature fusion. The survey also reviews supervision strategies for limited 3D data and ambiguity between 2D observations and 3D poses.

  • Frameworks: Representative single-person frameworks use volumetric heatmaps, lifting 2D poses to 3D, or fusing image features with 2D poses.The survey separately highlights approaches for limited 3D data and inherent 2D-to-3D ambiguity.
  • Volumetric Heatmaps: Heatmap-based methods represent each 3D keypoint as a volumetric Gaussian distribution and recover coordinates from local maxima.These methods directly estimate volumetric heatmaps from monocular images.
  • 2D-to-3D Lifting: Lifting methods first estimate 2D poses from images and then regress their corresponding 3D poses.This decomposition reduces model complexity and can improve generalization; Simple-3D uses two fully connected blocks to estimate keypoint depth.
  • 2D-to-3D Lifting: Pose matching can recover 3D poses by searching a library of 2D-3D pairs generated through random-camera projections.Other methods refine intermediate 2D poses using 3D back-projection or learn lifting through cycle consistency.
  • Feature Fusion: Fusing image features with 2D poses adds contextual information that can help resolve ambiguities left by sparse 2D keypoints.The cited approaches integrate local image texture features with pose representations.
  • Limited 3D Data: Unsupervised and weakly supervised approaches reduce dependence on fully annotated 3D datasets using weak labels, multi-view consistency, or synthetic data.The survey describes ordinal depth relations, multi-view supervision, and image-stitching or 3D-projection pipelines.

5) Solving the Inherent Ambiguity Problem:

Monocular 3D estimation is intrinsically ambiguous because multiple 3D poses can project to the same 2D pose. Methods address this through temporal, structural, geometric, and scene-level representations across skeleton and mesh paradigms.

  • Ambiguity: A single 2D pose may correspond to multiple 3D poses, making depth recovery ambiguous, especially for 2D-to-3D lifting.Temporal consistency and dynamics are among the priors used to select plausible 3D poses.
  • Temporal and Body Priors: Temporal models address ambiguity by using spatio-temporal inputs, motion compensation, and body-prior constraints.Examples model large body shifts between frames or encode kinematics, symmetry, and driven joint coordination.
  • Structural Representations: Hierarchical bone representations reduce ambiguity by explicitly modeling geometric dependence through bone lengths and joint directions.The representation is designed to remain closer to body structure than unconstrained joint coordinates.
  • Multi-person Scenes: Multi-person 3D estimation must recover each person’s root depth and relative keypoint depths, commonly through top-down or bottom-up processing.Top-down methods detect people before estimating their 3D poses; bottom-up methods group detected body parts.
  • Multi-person Scenes: Top-down methods estimate poses from detected person regions, while bottom-up methods infer keypoints and associate them into individual skeletons.LCR-Net++ ranks pose candidates from anchor proposals, whereas bottom-up systems predict volumetric heatmaps, connections, and person groupings.
  • Mesh-based Representations: Mesh-based methods either regress mesh vertices directly or estimate parameters of statistical body models such as SMPL.SMPL-based systems derive body meshes and keypoints from estimated camera, pose, and shape parameters.

1) Solving the Lack of Data:

The survey reviews strategies for learning 3D pose and shape despite scarce annotations, including unpaired supervision, temporal or dense cues, optimization in the loop, and structured mesh representations. Multi-person mesh recovery additionally addresses occlusion, truncation, collisions, and depth ordering.

  • Limited Data: Unpaired 2D pose and 3D motion-capture data can supervise human mesh recovery when depth-wise annotations are unavailable.HMR combines 2D pose data with motion-capture supervision to avoid unreasonable 3D pose and shape.
  • Temporal Supervision: Temporal dynamics, motion hallucination, and smoothness provide additional supervision for estimating 3D pose from image sequences or static images.These methods learn current, past, and future poses or synthesize motion-related features.
  • Detailed Supervision: Multi-task and detailed supervision use DensePose, 2D and 3D keypoints, body parts, silhouettes, and pixel-level information to refine mesh reconstruction.HoloPose jointly estimates several cues and iteratively improves alignment between model-based estimates and DensePose.
  • Optimization-based Supervision: SMPL optimization in the training loop combines regression-based prediction with optimization-based refinement to provide additional 3D supervision.The method uses SMPLify during training to refine estimated results.
  • Mesh Representations: Structured mesh representations, including graph-based and dense body-part representations, address ambiguity in regressing complex meshes from global image features.GraphCMR estimates mesh-vertex locations with graph convolutions before recovering SMPL parameters.
  • Multi-person Mesh Recovery: Multi-person mesh recovery must handle truncation, environmental occlusion, and person-person occlusion beyond single-person estimation.Recent approaches extend recovery to multiple interacting or overlapping body meshes.
  • Multi-person Mesh Recovery: CenterHMR predicts body centers and mesh parameters in a bottom-up single-shot model, improving robustness under occlusion and crowded overlap.Its occlusion-aware center representation uses visible body-part features, while differentiable interpolation discourages mesh collisions.
  • Summary: 2D and 3D pose estimation, unsupervised or weak supervision, and improved 3D-body representations together advance skeleton- and mesh-based methods.The survey concludes that these strategies address limited 3D data to some extent and support more accurate, efficient mesh estimation.

1) Evaluation Metrics of 2D Pose Estimation:

2D pose estimation uses complementary metrics for body parts, keypoints, and detection quality. These measures differ in matching thresholds, normalization, and whether they penalize missed detections or false positives.

  • Part-based metrics: PCP measures body-part accuracy when both limb endpoints fall within 50% of their ground-truth locations.PCPm uses 50% of the mean ground-truth segment length as its matching threshold, but foreshortening can distort measurement.
  • Keypoint metrics: PCK measures keypoint accuracy using a threshold defined as a fraction of person bounding-box size.PCKh@0.5 instead uses 50% of head-segment length, while AUC aggregates performance across thresholds.
  • Detection metrics: AP evaluates pose detection without annotated test-time bounding boxes and penalizes both missed detections and false positives.For multi-person estimation, AP is computed from Object Keypoint Similarity between predictions and ground truth.
  • Detection metrics: OKS measures prediction-to-ground-truth similarity using keypoint distance, visibility, person scale, and per-keypoint falloff.Each keypoint’s OKS ranges from 0 to 1, and varying OKS thresholds produces precision-recall curves for AP and AR.
  • 3D metrics: 3D pose evaluation commonly reports MPJPE, PA-MPJPE, 3D PCK, AUC, MPJAE, and PA-MPJAE.MPJPE measures pelvis-aligned Euclidean error, whereas Procrustes-aligned metrics remove translation, rotation, and scale effects.

B. Datasets

Public datasets supply training data and standardized comparisons that have accelerated deep-learning-based pose estimation. Representative datasets are generally large, well annotated, and cover varied poses and scenes.

  • Dataset role: Dataset development has boosted deep-learning pose estimation by providing training sources and fair comparisons.The survey emphasizes dataset scale, pose diversity, scene diversity, and annotation quality when introducing representative datasets.

1) 2D Pose Datasets:

2D pose datasets span image- and video-level settings, single- and multi-person scenes, and increasingly difficult occlusion, crowding, and temporal-variation conditions. They support both method training and standardized evaluation across diverse environments.

  • Dataset organization: The survey organizes 2D datasets by image versus video level and single-person versus multi-person coverage.Examples and dataset summaries are presented in Table III and Fig. 11.
  • Image-level datasets: Image-level single-person datasets range from the small LSP benchmark to FLIC and the diverse MPII dataset.LSP contains 2,000 images, FLIC contains 5,003 movie images, and MPII spans 3,913 videos and 491 activities.
  • Image-level datasets: Image-level multi-person datasets include COCO, AI-Challenger, and CrowdPose, with annotations designed for keypoint detection and crowded scenes.COCO provides over 200K images and 250K person instances, while CrowdPose selects 20K crowd-focused images with about 80K annotated persons.
  • Benchmark results: COCO and bottom-up pose methods have become important benchmarks for multi-person estimation, with HigherHRNet reaching 70.5 mAP.The cited comparison states that bottom-up methods may achieve performance comparable to top-down approaches.
  • Video-level datasets: Video-level single-person datasets provide temporal pose annotations across actions, viewpoints, appearances, and natural backgrounds.J-HMDB contains 928 clips across 21 actions, while Penn Action contains 2,326 clips covering 15 actions.
  • Video-level datasets: PoseTrack supports multi-person pose estimation and tracking with keypoints, person identities, and head bounding boxes under occlusion and truncation.Its 2017 release contains 550 videos and 153,615 pose labels; top-down methods perform better, while bottom-up methods are more efficient.

2) 3D Pose Datasets:

3D pose benchmarks include controlled single-person datasets, in-the-wild multi-person datasets, synthetic data, and motion-capture resources. Their coverage and pose-space distributions expose a central limitation: many benchmarks remain biased toward common standing and walking poses.

  • Dataset organization: The survey groups 3D resources into single-person, multi-person, and benchmark-analysis categories.Table VII summarizes widely used 3D pose datasets, while Tables VIII and IX compare pose estimation and mesh recovery methods.
  • Single-person datasets: Human3.6M is a large indoor multi-view benchmark with 3.6 million poses across 15 scenarios and standardized subject-based evaluation protocols.Only seven subjects’ data is currently available because of privacy concerns.
  • Single-person datasets: HumanEva-I, MPI-INF-3DHP, and MoVi extend controlled 3D data with multiple actions, viewpoints, actors, and motion-capture annotations.MPI-INF-3DHP contains over 1.3M frames, while MoVi includes 60 female and 30 male subjects.
  • Synthetic and motion datasets: SURREAL offers large-scale synthetic data, but limited low-resolution textures make its rendered images unrealistic.AMASS instead unifies 15 motion-capture datasets into more than 110K motions for learning human motion priors.
  • Multi-person datasets: 3DPW provides single-view, in-the-wild multi-person sequences with 2D/3D poses, body scans, and SMPL parameters.It contains 60 sequences across activities and environments, but crowded scenes may label only the target person.
  • Multi-person datasets: CMU Panoptic and JTA support multi-person 3D evaluation through multiview capture and photorealistic synthetic urban videos.CMU Panoptic contains 1.5 million 3D skeletons, while JTA contains 512 videos recorded at 30 FPS.
  • Benchmark analysis: 3D pose methods perform best on indoor single-person benchmarks, while mesh-recovery methods show promising generalization on 3DPW.The survey also reports that mesh recovery is suited to comprehensive 3D human analysis and visualization.
  • Pose-space analysis: Pose-space analysis aligns annotations, clusters them with K-means, and visualizes them with UMAP across Human3.6M, 3DPW, MoVi, and MPI-INF-3DHP.The resulting distributions are uneven, with most samples near walking or standing postures, limiting dataset diversity.

VI. CONCLUSION AND FUTURE DIRECTIONS.

The survey finds substantial progress in deep learning-based monocular human pose estimation while identifying persistent challenges across data, realistic scenes, 3D evaluation, and interaction-aware understanding.

  • The survey covers over 200 deep learning-based studies spanning monocular 2D and 3D pose estimation.
  • Complex postures and crowded scenes remain difficult because existing models struggle with rare poses, rapid motion, occlusion, and unreasonable 3D reconstructions.
  • Scene-person relations could help infer invisible or occluded body parts in challenging real-world settings.
  • Monocular 3D mesh recovery lacks large-scale mesh datasets and dedicated protocols, making skeleton-based metrics insufficient for evaluating richer appearance information.
  • Realistic whole-body recovery is limited by scarce paired 3D datasets containing coordinated body, face, hand, hair, and clothing details.
  • Multi-person 3D estimation remains complicated by reliance on 2D detection and insufficient use of person-person or person-scene interactions.
Loading 2104.11536v1…