Source-linked AI summary
Detailed Human Shape Estimation from a Single Image by Hierarchical Mesh Deformation
Hao Zhu, Xinxin Zuo, Sen Wang, Xun Cao, Ruigang Yang
TL;DR
Recovering detailed 3D human shape from a single image remains difficult because existing parametric and volumetric representations lack sufficient expressiveness for clothing and surface detail. The paper introduces Hierarchical Mesh Deformation, which refines an SMPL-based mesh through coarse-to-fine free-form deformation using joints, silhouettes, and shading information. Experiments report significant quantitative and qualitative improvements, while pose ambiguity and depth-direction errors remain limitations.
Problem
Existing parametric and volumetric approaches lack sufficient expressiveness to represent detailed human shape, clothing, and the body's inherent complexity.
Method
HMD starts from an SMPL-based human mesh and applies successive coarse-to-fine residual deformations using hierarchical networks supervised by joints, silhouettes, and shading information.
Results
The method achieves significant improvements over state-of-the-art approaches in quantitative and qualitative assessments, including more accurate and detailed recovered bodies.
Takeaways & Limitations
Free-form deformation beyond a parametric human model enables detailed single-image shape recovery with good generalization to images in the wild.
Takeaways & Limitations
Pose ambiguities remain unresolved, with large predicted-mesh errors especially in the depth direction.
Abstract
from arXiv · showhide
This paper presents a novel framework to recover detailed human body shapes from a single image. It is a challenging task due to factors such as variations in human shapes, body poses, and viewpoints. Prior methods typically attempt to recover the human body shape using a parametric based template that lacks the surface details. As such the resulting body shape appears to be without clothing. In this paper, we propose a novel learning-based framework that combines the robustness of parametric model with the flexibility of free-form 3D deformation. We use the deep neural networks to refine the 3D shape in a Hierarchical Mesh Deformation (HMD) framework, utilizing the constraints from body joints, silhouettes, and per-pixel shading information. We are able to restore detailed human body shapes beyond skinned models. Experiments demonstrate that our method has outperformed previous state-of-the-art approaches, achieving better accuracy in terms of both 2D IoU number and 3D metric distance. The code is available in https://github.com/zhuhao-nju/hmd.git
1. Introduction
Recovering detailed 3D human shape from one image is difficult because existing parametric and volumetric representations lack sufficient expressiveness for clothing and surface detail. HMD combines a parametric initialization with hierarchical free-form deformation guided by image-derived constraints.
- Parametric models produce low-fidelity shapes that cannot capture clothing details, while volumetric CNN representations remain coarse.
- Compared with HMR, HMD recovers surface details that better fit the input image.
- Human-shape variation and pose make prior representations insufficiently expressive for the body's inherent complexity.
- HMD initializes an SMPL-based mesh and refines it through non-rigid 3D deformation in a coarse-to-fine framework.
- Window-cropped images improve deformation prediction accuracy and robustness, while a photometric term enables recovery of high-frequency details.
- The framework uses a project-predict-deform strategy and hierarchical updates incorporating body joints, silhouettes, and photometric stereo.
2. Related Work
Prior work represents human shape with either parametric generative models or direct non-parametric predictions. Despite progress, these approaches generally lack sufficient accuracy and detail, producing shapes that visually resemble undressed bodies.
- Previous approaches fall into parametric and non-parametric categories based on how they represent the human body.
- Parametric methods predict parameters of pretrained generative models such as SCAPE or SMPL from monocular images or silhouettes.
- Several parametric methods replace manual intervention with learned networks or decompose parameter prediction into specialized subnetworks.
- Non-parametric methods directly predict shape representations such as depth maps and body-part segmentations from images.
- Despite significant progress, prior methods still lack accuracy and detail, and their recovered shapes visually resemble undressed bodies.
3. Hierarchical Deformation Framework
The framework refines an initial SMPL mesh through hierarchical, coarse-to-fine deformation driven by image evidence. It combines joint, anchor, and per-vertex stages with silhouette, shading, and photometric cues to recover detailed human shapes.
- Framework overview: The framework begins with an estimated SMPL mesh, followed by three refinement stages that progressively predict mesh deformations.The refinement phases are arranged from coarse to fine and are trained successively using residual deformation from the preceding phase.
- Framework overview: Each stage projects the current 3D mesh into 2D, combines it with the source image, predicts updates, and deforms the mesh.The architecture proceeds through joint, anchor, and per-vertex levels.
- Deformation strategy: The method uses Laplacian mesh deformation to propagate predicted handle motions while preserving local geometry as much as possible.Handle motion is predicted from 2D joints and silhouettes, then applied to the 3D mesh through Laplacian deformation.
- Handle definitions: Joint handles represent ten body joints, while 200 anchor handles are selected across the template and constrained to move along surface normals.Joint motions are two-dimensional image-plane vectors; anchor selection uses position and surface normal clustering, excluding high-frequency facial and extremity regions.
- Handle definitions: The vertex-level mesh is subdivided to 27,554 vertices, enabling finer deformation than the sparse SMPL mesh.All subdivided vertices are treated as handles for detailed deformation.
- Image and shading cues: Cropped handle-centered inputs focus prediction networks on relevant image regions, while shading refinement adds surface details from color and depth information.Joint and anchor networks use cropped patches, and a Shading-Net predicts refined depth maps from color images and coarse depth maps.
4. Experiment
The experiments evaluate HMD on WILD, RECON, and SYN datasets using silhouette, joint, and 3D shape metrics. Results show progressively refined meshes, strong comparisons with prior methods, improved silhouette-conditioned predictions, and limitations under ambiguous viewpoints and difficult poses.
- Datasets: The evaluation uses WILD for images with annotated 2D joints and segmentations, plus RECON and SYN datasets with ground-truth 3D meshes.RECON contains 25 reconstructed human meshes, while SYN contains 300 synthetic human mesh models.
- Staging analysis: HMD progressively refines shape through joint, anchor, and vertex deformation, with silhouette supervision fitting the body and vertex deformation recovering details.The final vertex-deformed stage is reported as HMD-vertex.
- Ablation study: Window-cropped input improves silhouette IoU and joint error over full-image input while using a network that is 41% of the full-image model size.The ablation attributes the comparison to full-image and window-cropped inputs.
- Ablation study: Combining joint and anchor deformation achieves the best ablation performance, and adding ground-truth silhouettes improves all reported metrics.The silhouette-conditioned variant is labeled HMD (+Sil.).
- Comparison with other methods: Compared with SMPL-based methods, HMD achieves the best performance across all metrics on the three datasets, while its RECON 3D error is comparable to BodyNet.BodyNet can omit limbs that are not visible, whereas HMD recovers a complete body model; BodyNet performs worse on SYN for slim subjects.
- 3D error analysis: Single-view depth ambiguity limits 3D accuracy: meshes can fit the input image closely while differing substantially from ground truth in side view.This explains why improvements in 2D metrics are relatively larger than improvements in 3D metrics.
- View synthesis: Texture mapping produces better side-view textured models when the recovered mesh matches the image, and foreground dilution further improves marginal texture regions.Figure 7 marks bad, defective, and fine regions with red, yellow, and green boxes.
5. Conclusion
The proposed coarse-to-fine framework combines SMPL-based recovery with free-form deformation under joint, silhouette, and shading supervision to reconstruct more accurate and detailed human bodies. However, pose ambiguities remain unresolved, with large mesh errors especially in depth.
- The method reconstructs detailed human body shapes in a coarse-to-fine manner by adding free-form deformations to an SMPL-based recovery process.
- A hierarchical framework uses joints, silhouettes, and shading information to restore more accurate and detailed human bodies.
- Pose ambiguities remain unresolved, and predicted body meshes still contain large errors, especially in the depth direction.