PaperlayerView exact revision on arXiv

Source-linked AI summary

Recovering 3D Human Mesh from Monocular Images: A Survey

Yating Tian, Hongwen Zhang, Yebin Liu, Limin Wang

arXiv:2203.01923v6cs.CVcs.GR

TL;DR

Monocular 3D human mesh recovery seeks detailed, physically plausible human geometry from images despite ambiguity, occlusion, and limited annotations. This survey organizes recovery methods, body models, training objectives, datasets, metrics, benchmarks, and future directions across the field.

  • Problem

    Monocular images provide insufficient information for uniquely recovering detailed 3D human pose and shape, while robust, accurate, and efficient mesh recovery remains challenging.

  • Method

    The survey categorizes monocular mesh recovery by design paradigm, reconstruction granularity, and application scenario, while reviewing body models, frameworks, objectives, datasets, metrics, and physical-plausibility considerations.

  • Results

    The survey provides a comprehensive overview of methods from the past decade and compares their datasets, evaluation metrics, and benchmark performance.

  • Takeaways & Limitations

    The synthesis identifies optimization and regression-based recovery, whole-body reconstruction, weak or unsupervised learning, and physical plausibility as central directions for advancing the field.

  • Takeaways & Limitations

    The review focuses on monocular RGB images or videos and excludes RGBD, multi-view inputs, clothing modeling, and neural rendering.

Abstract

from arXiv · show

Estimating human pose and shape from monocular images is a long-standing problem in computer vision. Since the release of statistical body models, 3D human mesh recovery has been drawing broader attention. With the same goal of obtaining well-aligned and physically plausible mesh results, two paradigms have been developed to overcome challenges in the 2D-to-3D lifting process: i) an optimization-based paradigm, where different data terms and regularization terms are exploited as optimization objectives; and ii) a regression-based paradigm, where deep learning techniques are embraced to solve the problem in an end-to-end fashion. Meanwhile, continuous efforts are devoted to improving the quality of 3D mesh labels for a wide range of datasets. Though remarkable progress has been achieved in the past decade, the task is still challenging due to flexible body motions, diverse appearances, complex environments, and insufficient in-the-wild annotations. To the best of our knowledge, this is the first survey that focuses on the task of monocular 3D human mesh recovery. We start with the introduction of body models and then elaborate recovery frameworks and training objectives by providing in-depth analyses of their strengths and weaknesses. We also summarize datasets, evaluation metrics, and benchmark results. Open issues and future directions are discussed in the end, hoping to motivate researchers and facilitate their research in this area. A regularly updated project page can be found at https://github.com/tinatiansjz/hmr-survey.

1 INTRODUCTION

Monocular 3D human mesh recovery extends 2D understanding toward detailed, physically plausible representations of pose, shape, contact, gesture, and expression. This survey reviews its applications, challenges, paradigms, scope, and organization.

  • Motivation: Complex human behaviors require detailed body descriptions beyond 2D keypoints, silhouettes, part segmentations, or sparse 3D joints.Surface shape, contact, gesture, and expression matter because humans interact with the world through their skin rather than unobserved joints.
  • Motivation: Statistical models, especially SMPL and SMPL-X, helped shift research toward 3D mesh recovery and expressive body, face, and hand reconstruction.Their annual citations grew rapidly after SMPL’s 2015 release and SMPL-X’s 2019 release.
  • Applications: Human mesh recovery supports applications including VR/AR content creation, virtual try-on, computer-assisted coaching, fitness games, and dynamic simulation.The survey illustrates these applications through examples involving fitness, diving, and swimming.
  • Challenges: The task remains difficult because lifting 2D observations into 3D is ambiguous, body kinematics are flexible, environments create intersections, and annotated 3D data are insufficient.The community still faces challenges in achieving robust, accurate, and efficient recovery.
  • Scope and organization: The survey covers monocular RGB images and videos, single- and multi-person recovery, statistical body models, methods, datasets, metrics, benchmarks, and future directions.RGBD and multi-view inputs, clothing modeling, neural rendering, and appearance-focused reconstruction are outside its scope.

2 HUMAN MODELING

Human modeling progressed from geometric primitives and scan-based representations to statistical mesh models that encode pose and identity-dependent shape. SMPL became a widely used body template, while later extensions address hands, face, soft tissue, and limitations in deformation modeling.

  • Geometric and scan-based modeling: Early body models approximated humans with geometric primitives such as rectangles, cylinders, ellipsoids, superquadrics, metaballs, and graphical models.These representations preceded statistical models derived from scanned bodies.
  • Geometric and scan-based modeling: Full-body scans provide detailed surface measurements but require template registration, skeleton fitting, and skinning to produce watertight, animatable meshes.The preprocessing pipeline converts dense point clouds and triangulated scans into structured human body meshes.
  • Statistical body modeling: Statistical body models learn from extensive 3D scans and factor body deformation into identity-dependent and pose-dependent components.SCAPE and SMPL are representative models that omit hand articulation and facial expression in their basic forms.
  • SMPL: SMPL maps pose parameters θ and shape parameters β to a triangulated mesh with 6,890 vertices and uses a 23-joint skeletal rig.Its low-dimensional shape coefficients are obtained through PCA, while pose parameters encode relative joint rotations.
  • SMPL: SMPL first generates a rest pose with pose- and shape-dependent corrective blend shapes, then applies linear blend skinning to pose the mesh.The corrective blend shapes address standard linear blend skinning artifacts, and the model can be extended with dynamic soft-tissue deformations.
  • Model limitations and extensions: SMPL has limitations including spurious long-range correlations, missed shape–pose correlations, and difficulty representing highly nonlinear soft-tissue deformations.STAR, SoftSMPL, and implicit or explicit learning-based models seek greater descriptive capability.
  • Whole-body modeling: The SMPL family expanded to hands and face through models such as MANO, SMPL+H, and SMPL-X, with SMPL-X jointly modeling body, face, and hands.These extensions support increasingly expressive whole-body reconstruction.

3 HUMAN MESH RECOVERY

Human mesh recovery is organized around statistical body models and two complementary paradigms: iterative optimization that fits models to 2D observations, and regression that predicts mesh representations from pixels. The survey compares their representations, objectives, architectures, and extensions to whole-body recovery.

  • Body Models: Statistical body models support monocular estimation of human body pose and shape, with SMPL becoming a prevailing model for 3D body recovery.Their open-source availability, annotation methods, datasets, and surrounding research community contributed to SMPL’s adoption.
  • Optimization-based Paradigm: Optimization-based methods fit parametric body models iteratively to 2D observations using alignment data terms and regularizers favoring physically plausible poses.SMPLify, for example, combines projected-joint alignment with pose, shape, and interpenetration penalties.
  • Optimization-based Paradigm: Deep learning can strengthen optimization by providing learned priors, as EFT optimizes near a pretrained regressor without external regularization terms.EFT reinitializes the regressor for each new sample after fitting is completed.
  • Regression-based Paradigm: Regression-based methods process image pixels directly and predict parametric or non-parametric mesh representations through deep neural networks.The survey reviews output types, intermediate representations, supervision in 2D and 3D, and network architectures.
  • Regression-based Paradigm: Parametric outputs regress low-dimensional pose and shape vectors, whereas non-parametric outputs relax reliance on a predefined parameter space while retaining mesh topology.UV position maps recast vertex prediction as image-to-image translation suited to convolutional networks.
  • Training Objectives: Weakly supervised and unsupervised regression use reprojection-and-compare or render-and-compare strategies when 3D annotations are unavailable.The most common 2D supervision target is 2D joints, obtained using predicted camera parameters.
  • Whole-Body Recovery: Whole-body recovery extends body reconstruction to hands and face by combining part-specific features, regressors, models, and datasets.Representative systems exploit inter-part relationships, feature fusion, synthetic data, and adaptive integration of conflicting subnetwork outputs.

4 MULTI-PERSON RECOVERY

Multi-person recovery is divided into top-down and bottom-up strategies. Top-down methods reuse single-person pipelines after detection, whereas bottom-up methods preserve a holistic view for computationally efficient joint recovery.

  • Top-Down Strategy: Top-down recovery reduces multi-person estimation to single-person recovery by feeding cropped detections into existing single-person networks.Its main drawbacks are repeated feature extraction and a limited receptive field within each bounding box.
  • Bottom-Up Strategy: Bottom-up recovery predicts multiple people jointly from a holistic image view, using one-stage designs that can be computationally efficient.Single-shot methods represent instances by body-center points and predict localization heatmaps alongside mesh parameter maps.
  • Human-Human Interactions: Interaction-aware methods add collision or contact constraints to prevent overlapping human models and represent human-human interactions.Examples include volume occupancy losses based on adaptive limb primitives and facet-level interaction-contact losses.

5 RECOVERY FROM MONOCULAR VIDEOS

Monocular video recovery must address occlusions and motion jitters that arise when frames are processed independently. The survey reviews temporal encoding, decoding, supervision, adaptation, and global-coordinate strategies for improving consistency.

  • Motivation: Frame-independent image methods are prone to occlusions and motion jitters because they lack temporal constraints.Video methods therefore seek consistency in both spatial and temporal feature encoding.
  • Temporal Encoding: Temporal encoding commonly uses convolutional or recurrent networks to combine static image features with motion cues across frames.Examples include optical flow, 2D heatmaps, single-frame ConvNets, LSTMs, and pretrained backbones.
  • Temporal Decoding: Decoding strategies and optimization objectives reduce jitter by refining frame parameters, modeling dynamics, or learning smooth motion subspaces.HMMR predicts pose changes, while MEVA uses a VAE to generate coarse but smooth motions before retrieving finer motions.
  • Temporal Supervision: Video supervision includes motion reprojection errors, velocity priors, and adversarial training strategies applied across adjacent or shuffled frames.These objectives use temporal relationships beyond single-frame reconstruction.
  • Global Reconstruction: Some systems reconstruct meshes in world coordinates by combining camera motion, tracking, and human reconstruction for dynamic-camera videos.GLAMR extracts local motions, infills missing poses, predicts global trajectories, and jointly optimizes camera poses and global motions.

6 HUMAN-SCENE INTERACTIONS

Human-scene recovery treats people, scenes, and interacting objects jointly rather than reconstructing humans in isolation. Existing methods impose collision, contact, support, spatial-arrangement, scale, or depth constraints, but predefined contact candidates can limit generalization.

  • Motivation: Monocular human-scene methods jointly reason about humans, scenes, and objects to infer spatial arrangements and contacts in interactive scenarios.This extends isolated human reconstruction toward scene-aware understanding.
  • Existing Methods: HolisticMesh and PHOSA impose human-scene constraints including collision, contact, ground support, spatial arrangement, scale, and ordinal depth.Their losses operate at joint, instance, or part levels depending on the interaction being modeled.
  • Limitations: Predefined candidate contact vertices or pairs constrain the generalization of HolisticMesh and PHOSA to diverse scenes.Both methods depend on these manually specified interaction candidates.

7 PHYSICAL PLAUSIBILITY

Physically plausible mesh recovery requires more than alignment to 2D observations: camera models, contact constraints, and body priors help address visual artifacts and 2D-to-3D ambiguity.

  • Camera Models: Camera models are needed because monocular images do not provide exact perspective-camera intrinsics and extrinsics.A weak-perspective model commonly predicts scale and x/y translation, converting scale into z-axis translation.
  • Camera Models: Full-image camera estimation can address localization errors caused by processing only cropped images.Methods such as CLIFF use original-image information, while other approaches estimate focal length or camera parameters in world coordinates.
  • Contact Constraints: Contact constraints encourage proper contacts and penalize erroneous interpenetration between humans, scenes, or body parts.Ground-plane fitting, predicted contact status, and self-contact objectives are used to improve plausibility.
  • Human Priors: Diffusion-based priors can produce multiple hypotheses for occluded inputs because of their probabilistic nature.Temporal priors instead learn motion patterns using recurrent models, autoencoders, discriminators, or conditional VAEs.

8 DATASETS

Mesh-recovery datasets combine rendered data, MoCap captures, and pseudo-3D labels, each balancing annotation precision, realism, scale, and diversity.

  • Annotation Acquisition: Mesh labels are obtained from rendered models or scans, marker-based and marker-less MoCap, and image-fitting procedures for in-the-wild data.Rendered data are precise but can lack realism, while real-image annotation remains difficult.
  • Annotation Acquisition: Pseudo-labels for in-the-wild datasets are less accurate than MoCap labels but improve regression-method generalization through scale and diversity.CLIFF uses original-frame information to produce more accurate labels, especially for global rotations.
  • Rendered Datasets: Rendered datasets provide paired 3D annotations and diverse synthetic conditions, including body shape, motion, environments, and appearance.SURREAL, GTA-Human, AGORA, and BEDLAM illustrate increasing realism, scale, and diversity in synthetic data.
  • MoCap Datasets: MoCap datasets provide accurate 3D supervision across controlled studios, outdoor scenes, multi-person interactions, and occlusion scenarios.Examples include HumanEva, Human3.6M, 3DPW, CMU Panoptic, MPI-INF-3DHP, and 3DOH50K.
  • Interaction Datasets: Scene-interaction datasets extend mesh recovery toward human-object reconstruction using RGB-D, LiDAR, video, and IMU measurements.PROX, BEHAVE, and SLOPER4D target indoor, natural, and urban interaction settings.
  • Datasets with Pseudo 3D Labels: In-the-wild 2D pose datasets offer rich subject, pose, and scene diversity but lack 3D pose or mesh annotations.Researchers fit pseudo-ground-truth labels using optimization or regression networks.

9 EVALUATION

Evaluation uses multiple joint, angle, and mesh-recovery benchmarks, but reported values remain difficult to compare because protocols and training conditions vary.

  • Metrics: MPJPE measures average Euclidean 3D joint error after root matching, while W-MPJPE measures joint error in world coordinates.PA-MPJPE additionally applies rigid Procrustes alignment before computing the error.
  • Metrics: MPJAE measures 3D joint-orientation deviation using geodesic distance in SO(3), with PA-MPJAE applying Procrustes alignment.These metrics evaluate angular rather than positional error.
  • Body Recovery Benchmarks: Performance on Human3.6M and 3DPW has improved, but comparisons fluctuate with backbones, outputs, pseudo-label quality, datasets, training strategies, and protocols.SPIN established an evaluation protocol widely adopted by subsequent methods.
  • Whole-Body Benchmarks: Full-body recovery with face and hands remains much harder than body-only recovery, with substantially higher reconstruction error on AGORA and EHF.The reported evaluations include full-body, body, face, and hand results.

10 CONCLUSION AND FUTURE DIRECTIONS

The survey organizes a decade of monocular 3D human mesh recovery research and identifies persistent challenges in occlusion, temporal stability, scene interaction, supervision, group reconstruction, and whole-body modeling.

  • Conclusion: The survey categorizes methods by design paradigm, reconstruction granularity, and application scenario while reviewing physical plausibility, datasets, metrics, and benchmarks.It also highlights future directions for advancing the field.
  • Future Directions: Heavy occlusion still limits robustness and stability, motivating multiple plausible reconstructions or pose distributions when visual evidence is insufficient.Occlusions arise from self-overlap, nearby people, and scene objects.
  • Future Directions: Video methods remain vulnerable to motion jitters, especially under heavy occlusion or complex contexts, making long-term temporal refinement necessary.Low-pass filters and learning-based refinement networks are cited as approaches to improve temporal smoothness.
  • Future Directions: Scene-unaware methods can produce interpenetration, while physically consistent approaches commonly simplify scenes as flat floors.Future work is directed toward incorporating scene meshes for humans interacting with static environments.
  • Future Directions: 3D mesh datasets are costly to build and less diverse than 2D datasets, motivating greater use of abundant unlabeled data.MoCap setup, cleaning, and annotation create substantial data-construction demands.
  • Future Directions: Whole-body recovery is constrained by scarce whole-body datasets and the combined effects of occlusion, blur, depth ambiguity, and hand interactions.Separate body, hand, and face datasets are often used to compensate for incomplete whole-body data.
  • Future Directions: Clothing reconstruction remains limited by the representation power of minimally clothed parametric models and oversmoothed, pose-sensitive surfaces.Meshes, point clouds, and implicit fields have been explored, with hybrid representations proposed to combine their modeling strengths.
Loading 2203.01923v6…