PaperlayerView exact revision on arXiv

Source-linked AI summary

A review of 3D human pose estimation algorithms for markerless motion capture

Yann Desmarais, Denis Mottet, Pierre Slangen, Philippe Montesinos

arXiv:2010.06449v3cs.CV

TL;DR

Modern markerless 3D pose estimation has proliferated, making method choice difficult despite progress toward accurate motion capture. This review compares recent methods through metrics, benchmarks, and structural taxonomies, finding that 3D pose error decreased by nearly 70 mm over the last decade while no consensus on the best method has emerged.

  • Problem

    Markerless 3D methods must become competitive with classical and marker-based capture while supporting accurate motion reconstruction under challenging conditions.

  • Method

    The review analyzes recent pose-estimation methods through metrics, benchmarks, architectures, and a taxonomy based on accuracy, robustness, and speed.

  • Results

    3D human pose estimation error decreased by nearly 70 mm over the last decade, mainly due to convolutional networks.

  • Takeaways & Limitations

    No consensus has yet been established on the best method, motivating continued research into the 3D nature of the problem.

  • Takeaways & Limitations

    Robustness to fast motion and occlusions requires a higher acquisition frequency.

Abstract

from arXiv · show

Human pose estimation is a very active research field, stimulated by its important applications in robotics, entertainment or health and sports sciences, among others. Advances in convolutional networks triggered noticeable improvements in 2D pose estimation, leading modern 3D markerless motion capture techniques to an average error per joint of 20 mm. However, with the proliferation of methods, it is becoming increasingly difficult to make an informed choice. Here, we review the leading human pose estimation methods of the past five years, focusing on metrics, benchmarks and method structures. We propose a taxonomy based on accuracy, speed and robustness that we use to classify de methods and derive directions for future research.

1. Introduction

Markerless 3D pose estimation seeks accurate motion reconstruction from ordinary imagery while avoiding the constraints of classical capture. This review organizes recent methods around accuracy, speed, robustness, metrics, benchmarks, and architectural families to guide method selection.

  • 2D pose estimation surpassed 90% detection on all human joints, supported by convolutional networks and large-scale datasets.
  • 3D markerless methods aim to reconstruct individual motion from monocular images while tolerating severe occlusion.
  • Classical motion capture is precise but constrained by complex sensors and acquisition environments.
  • Its taxonomy emphasizes method families and evaluates accuracy, robustness, and speed to help developers, engineers, and researchers choose algorithms.
  • The review focuses on 3D pose estimation, including multi-view techniques and methods using sensors such as inertial measurement units.

2. Methods Evaluation

The review compares 3D pose-estimation methods across benchmarks using metrics for positional, angular, threshold-based, temporal, and surface accuracy. Each metric captures different evaluation priorities and has specific interpretive limitations.

  • Position-based metrics: MPJPE averages Euclidean distances between estimated and ground-truth joint coordinates and serves as a broad baseline metric.It can evaluate methods estimating coordinate positions and overall skeleton structure.
  • Temporal metrics: MPJVE applies MPJPE to the first derivative of pose sequences to measure prediction smoothness over time.It is useful for comparing models that use temporal data.
  • Angular metrics: MPJAE measures angular errors between joint segments, supporting analyses focused on limb angles such as rehabilitation or sport motion.Its local calculation can produce apparently low errors for globally misaligned skeletons, limiting perceptual interpretation.
  • Threshold metrics: Threshold metrics count correctly detected joints or limbs under predefined criteria, including PCP and PCK variants.Subject-specific thresholds reduce limb-size bias, whereas thresholding sacrifices sensitivity to millimeter-scale coordinate differences.
  • Surface metrics: Surface-oriented tasks use geodesic metrics such as GPS to compare estimated and ground-truth points on reconstructed body surfaces.

3. Architectures for Human Pose Estimation

The reviewed architectures are grouped into human-body-model, learning-based, and geometric-information methods. Learning approaches use backbone networks and task-specific loss functions, with taxonomy tables recording their structures and intermediate representations.

  • The main method families are human body models, learning algorithms, and geometric information.
  • Neural-network approaches employ backbone networks and create new loss functions for pose estimation.
  • Table 2 classifies the reviewed methods and records intermediate representations for multi-stage methods.
  • Common listed backbones include Stacked Hourglass, Cascaded Pyramid, High Resolution Network, and Simple Baselines.

Human Body Models

Human pose estimation methods incorporate skeletal, mesh, pictorial, multi-view, and volumetric body representations. These models impose structural information or exploit multiple views to infer 3D joint locations and body shape.

  • Part-based models: Pictorial Structure Models estimate poses by minimizing an energy function combining joint-location error with limb-length deformation penalties.
  • Skeleton models: Kinematic skeletons represent linked joints as vectors and impose angular and length constraints during pose detection.
  • Mesh models: Mesh models reconstruct the complete body surface, providing richer information than skeleton models and supporting virtual-scene representation or animation.SMPL is a frequently used mesh model adapted through pose and shape parameters.
  • Multi-view methods: Multi-view methods triangulate 3D joints from 2D coordinates or fuse features along epipolar lines before estimating pose.The reconstruction scheme depends on camera calibration and available intrinsic and extrinsic parameters.
  • Volumetric methods: Shape-from-silhouette methods segment humans across views and reconstruct volumetric representations that can serve as intermediate features for joint prediction.

Learning Approaches

The review surveys learning architectures for 3D markerless pose estimation, spanning convolutional, temporal, graph, recurrent, attention, and adversarial approaches. Backbone choices depend on data representation and modeling assumptions, while the field continues improving state-of-the-art performance.

  • Backbone architectures: The review covers common backbone principles and reports that simple baselines can achieve competitive results at efficient computational cost.The reviewed architectures include ResNet, stacked hourglass, cascaded pyramid, and high-resolution networks.
  • Convolutional approaches: 2D convolutions dominate monocular methods, whereas multi-view, video, and multimodal inputs motivate volumetric and temporal techniques.Multi-view features can feed 3D convolutional networks, and temporal convolutions use past and future frames.
  • Graph-based approaches: Graph neural networks encode pose location and temporal trajectory, including spectral convolutions based on graph-Laplacian eigenvalues.
  • Recurrent approaches: LSTM sequence-to-sequence models retain information from previous inputs through gated cell-state updates when processing pose sequences.
  • Attention mechanisms: Attention mechanisms select informative frames or exploit multi-view epipolar lines corresponding to joint points.
  • Adversarial learning: Adversarial learning improves pose consistency by penalizing generated 3D poses that conflict with known configurations.

4. Methods Review

The review compares leading markerless methods across monocular, temporal, multi-view, and multimodal settings. Accuracy has improved substantially, but no single best method has emerged across the diverse approaches and evaluation conditions.

  • Review scope: The review selects methods using state-of-the-art performance, citation impact, and recent or original approaches that may advance research.
  • Overall trends: Average error fell from about 100mm to less than 20mm within 10 years, demonstrating rapid accuracy improvement.
  • Overall trends: The review finds no consensus on the best method because approaches and modalities remain diverse.
  • Monocular methods: Monocular methods include direct 2D-to-3D lifting, weak supervision, multi-person single-shot prediction, and volumetric or heat-map-based architectures.
  • Monocular methods: 41.1mm average error is reported for the most accurate reviewed monocular non-temporal method on a single isolated RGB image.
  • Temporal methods: Temporal models use motion-aware designs, with one reviewed model identifying 5 frames as its optimal sequence length.
  • Multi-view and multimodal methods: 17.7mm MPJPE is reported as the best Human3.6M result for multi-view input, while inertial sensors complement video in multimodal systems.

5. Analysis and discussion

The discussion frames 3D pose estimation as an application-dependent problem whose requirements vary across accuracy, speed, and robustness. The review therefore reports these criteria separately to support comparisons for different use cases.

  • Applications: Pose estimation supports motion analysis, entertainment, human-computer interaction, security, robotics, and avatar or game control.
  • Application requirements: Different applications require different levels of accuracy, speed, and robustness, even within the same broad category.
  • Evaluation framework: The review describes each performance criterion separately and explains which use cases favor each criterion.
  • Evaluation framework: Accuracy is reported using MPJPE, robustness reflects the assumptions or constraints needed for correct detection, and speed is summarized by real-time capability.
  • Evaluation framework: The resulting tables enable cross-comparison of methods according to application-specific requirements.

5.1. Accuracy

Accuracy is the primary evaluation criterion, but application requirements vary from semantic pose interpretation to maximal measurement precision. Across configurations, multi-view methods achieve the lowest reported errors, while temporal and model-based approaches offer distinct accuracy advantages and trade-offs.

  • Evaluation limitations: Accuracy comparisons are constrained because marker-based reference systems can contain errors and older datasets may have low-resolution videos or inaccurate annotations.The review also notes that many methods remain limited to labels present in their training data.
  • Accuracy across configurations: 10mm is the reported average error gap between monocular and multi-view methods.Multi-view methods exploit geometric constraints but require camera calibration.
  • Accuracy across configurations: 20 mm errors are achievable when geometric scene knowledge is combined with learning-based optimization.The review notes higher errors in general contexts, including 34mm MPJPE when evaluated on a different dataset than training.
  • Monocular methods: 41.1mm MPJPE on Human3.6M is reported for monocular methods, which are the simplest because they require only a single image.Jointly optimizing a mesh model and neural network is described as effective for monocular pose estimation.
  • Method structures: Two-stage top-down algorithms achieve the best results across monocular image, monocular video, and multi-view benchmarks.Effective approaches include 2D-to-3D regression, parametric-model initialization, temporal sequences, occlusion-aware modules, generative models, volumetric inputs, and multi-view triangulation.
  • Temporal and multi-view methods: Video-sequence methods perform better for temporally coherent activities, whereas monocular methods better detect complex static poses.The review identifies simultaneous exploitation of temporal and multi-view data as an underexplored direction.

5.2. Robustness

Robustness varies with assumptions about cameras, subjects, motion, temporal context, hardware, and occlusion. The review argues that accuracy alone is insufficient because less constrained systems often trade against hardware, data, or temporal requirements.

  • Motion, subject, and occlusion constraints: Many state-of-the-art methods still assume a single person, limited motion, fixed cameras, or no severe occlusion.Fast motion, moving-camera footage, overlapping body parts, and multi-person scenes remain difficult settings.
  • Data and environment constraints: Generalization to in-the-wild situations remains difficult because large motion-capture datasets are still captured mainly in indoor studios.Data augmentation can create new scenes by changing backgrounds or lighting, but benchmark protocols are needed to assess generalization.
  • Temporal constraints: Temporal methods can be adapted to real-time use with only past frames, but accuracy decreases and sufficient footage may still be required.Methods differ in their temporal receptive fields; some accuracy saturates beyond 200 frames.
  • Overall conclusion: The review finds that robustness is often overlooked, while method complexity is rarely considered alongside accuracy.For temporal methods, those avoiding future frames perform best; methods addressing fast motion and occlusion may require higher acquisition frequency.

5.3. Speed

The review evaluates speed through inference rate, hardware, latency, and model complexity, while noting that reported measurements are not always end-to-end. Faster methods tend to have somewhat below-average accuracy.

  • Reported speed measures: 3ms per frame is reported by Hossain and Little (2018) and Martinez et al. (2017), but some methods report only per-frame inference time.The review notes that some reported fps values exclude the first-stage 2D detector.
  • Reported speed measures: 25 and 30 frames per second are reported for detection by Huang et al. (2019) and Mehta et al. (2017), respectively.These figures concern detection rather than necessarily complete motion-capture pipelines.
  • Model complexity: Deep architectures are often unsuitable for real-time applications because a network forward pass takes too long.Parameter counts and floating-point operations provide ways to examine model complexity.
  • Real-time performance: Frames per second and the GPU used are reported for real-time methods, but robustness varies and the fastest methods tend to have below-average accuracy.The review identifies Huang et al. (2019) as the most accurate real-time method in its collection.
  • Real-time performance: 37.5 MPJPE on Human3.6M is achieved by Huang et al. (2019) without its IMU component.The method also performs well on TotalCapture when IMU data is added.

5.4. Recommendations for users

Method choice should depend on the application’s priorities among accuracy, robustness, speed, simplicity, and environmental constraints. The review gives different recommendations for security, motion analysis, human–computer interaction, and entertainment.

  • Security: Security applications prioritize robustness and real-time speed over maximum precision because identifying movement semantics matters most.The review recommends robust real-time monocular or temporal methods, with Mehta et al. (2020) when multi-person detection is needed.
  • Motion analysis: Motion analysis typically operates offline in controlled environments, making accuracy the main criterion and reducing the importance of real-time performance.Multi-view configurations are reported to achieve less than 30 mm average error per joint.
  • Human–computer interface: When multiple cameras or calibration are unavailable, monocular temporal methods such as Kolotouros et al. (2019) or Cheng et al. (2020) can be considered.The recommendation addresses settings where multi-view acquisition is not possible.
  • Human–computer interface: For simplicity and usability, flexible systems based on strong 2D detectors, including Mathis and Mathis (2019) or Martinez et al. (2017), produce efficient results.This recommendation targets applications requiring easy-to-use systems.
  • Entertainment: Controlled entertainment settings emphasize realistic poses and accuracy, whereas real-time control in games or VR requires speed.Multi-view markerless systems can replace classical systems offline, while Huang et al. (2019) supports real-time operation in multi-view configurations.

5.5. Challenge for future research

Future research must improve 3D pose accuracy, temporal and spatial consistency, real-time efficiency, and robustness to occlusions and multi-person scenes.

  • About 40 mm average joint position error is reported for monocular and temporal approaches, versus about 20–30 mm for multi-view approaches.
  • Monocular methods remain limited by inconsistent detection across space and time, while accuracy differences among joints and motion jitter require attention.
  • Approximately 90% of joints accurately detected is identified as a future goal, compared with about 85% average 3DPCK on MPI-based evaluations.
  • Multi-view systems can perform close to the best monocular methods despite access to epipolar geometry and 3D information.
  • In controlled environments, combining pose estimation with IMU or depth sensors could provide better multimodal information.
  • Real-time estimation on average commercial equipment remains challenging because many proposals rely on multi-stage computations.
  • Strong assumptions about occlusions and multi-person configurations still prevent application to arbitrary videos or images.

6. Conclusion

The conclusion reviews a diverse field whose methods commonly use 2D detection as an intermediate step. It identifies richer data, computational parsimony, ease of use, and more direct 3D reasoning as future priorities.

  • Deep learning has improved computer vision, but the field has not yet established a consensus on human pose estimation methods.
  • Current techniques use temporal convolutional networks, 3D human body models, or learnable triangulation across different input types.
  • 2D detection is a common intermediate step across these approaches.
  • Richer datasets containing varied poses, movements, and contexts are considered essential for robustness.
  • Computational parsimony could support real-time operation, less expensive hardware, and reduced power consumption.
  • Ease of use is identified as important for widespread adoption and for expanding algorithmic use cases.
  • Future approaches may reason directly about 3D structure, use less constrained multi-view systems with fewer cameras, or transfer temporal methods to multi-view settings.
Loading 2010.06449v3…