Source-linked AI summary
Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representation
Sun Yu, Ye Yun, Liu Wu, Gao Wenpeng, Fu YiLi, Mei Tao
TL;DR
Monocular 3D human mesh recovery must infer detailed pose and shape from ambiguous images, while prior approaches couple these factors in shared features. The paper decouples skeleton and mesh details spatially with DSD, models temporal coherence with SATN and order-recovery training, and reports improved benchmark performance.
Problem
Monocular mesh recovery must estimate detailed 3D shape and joint angles from ambiguous images, while existing methods derive pose, shape, and camera parameters from a coupled feature.
Method
An end-to-end framework uses a pluggable DSD module for skeleton-detail disentangling, SATN for temporal coherence, and shuffled-frame order recovery for motion dynamics.
Results
15.4% MPJPE and 23.8% PA-MPJPE improvements over state-of-the-art mesh methods are reported on Human3.6M, with state-of-the-art 3DPW results without fine-tuning.
Takeaways & Limitations
Skeleton-disentangled representation is reported as critical for better temporal motion modeling and generalization.
Takeaways & Limitations
The framework adopts a weak-perspective camera model for mapping 3D space to the 2D image plane.
Abstract
from arXiv · showhide
We describe an end-to-end method for recovering 3D human body mesh from single images and monocular videos. Different from the existing methods try to obtain all the complex 3D pose, shape, and camera parameters from one coupling feature, we propose a skeleton-disentangling based framework, which divides this task into multi-level spatial and temporal granularity in a decoupling manner. In spatial, we propose an effective and pluggable "disentangling the skeleton from the details" (DSD) module. It reduces the complexity and decouples the skeleton, which lays a good foundation for temporal modeling. In temporal, the self-attention based temporal convolution network is proposed to efficiently exploit the short and long-term temporal cues. Furthermore, an unsupervised adversarial training strategy, temporal shuffles and order recovery, is designed to promote the learning of motion dynamics. The proposed method outperforms the state-of-the-art 3D human mesh recovery methods by 15.4% MPJPE and 23.8% PA-MPJPE on Human3.6M. State-of-the-art results are also achieved on the 3D pose in the wild (3DPW) dataset without any fine-tuning. Especially, ablation studies demonstrate that skeleton-disentangled representation is crucial for better temporal modeling and generalization.
1. Introduction
The paper frames monocular 3D human mesh recovery as a complex, ambiguous problem and proposes decoupled spatial and temporal modeling to improve recovery and generalization.
- Motivation: Monocular mesh recovery estimates detailed 3D shape and joint angles, controlling more than 85 parameters and 6890 mesh vertices.This is more complex than predicting 14/17 skeleton joints.
- Motivation: Information loss, inherent ambiguity, and complex pose and shape changes make monocular 3D mesh recovery a frontier challenge.The same 2D image can correspond to multiple 3D poses and shapes.
- Motivation: Existing end-to-end methods couple pose, shape, and camera prediction through one feature, making predictions unstable and limiting generalization.Constrained datasets and insufficient 2D-pose supervision further expose the difficulty of recovering detailed mesh information.
- Approach: The lightweight, pluggable DSD module separates skeleton information from mesh details using independent extraction and bilinear aggregation.It is trained end-to-end while preserving factor decoupling in the new feature space.
- Approach: SATN combines self-attention with TCN to model short- and long-term temporal coherence, while shuffled-frame order recovery supplies unsupervised motion supervision.The self-attention module establishes temporal associations before TCN processing.
- Results: 15.4% MPJPE and 23.8% PA-MPJPE improvements over state-of-the-art mesh methods are reported on Human3.6M, with state-of-the-art 3DPW results without fine-tuning.Using DSD-reorganized features also relatively improves PA-MPJPE by 11.1% on 3DPW and 27.5% on Human3.6M.
2. Related Work
Prior work includes two-stage and direct single-image mesh recovery, temporal networks for monocular pose, and limited video-mesh methods; this paper targets coupling and temporal-supervision gaps.
- Single-image mesh recovery: ConvNet-based single-image mesh methods generally estimate parameters of the SMPL statistical human body model.They are divided into two-stage and direct-estimation categories.
- Single-image mesh recovery: Two-stage methods predict intermediate representations before SMPL parameters, improving domain-shift robustness but discarding important human-body details.Examples use parsing, silhouettes, and keypoint locations as intermediate inputs.
- Single-image mesh recovery: Direct end-to-end methods such as HMR map images to SMPL parameters and substantially outperform earlier two-stage methods.HMR uses adversarial discrimination of predicted SMPL parameters and can train with only 2D pose annotations.
- Single-image mesh recovery: The proposed DSD module addresses coupling by separating skeleton and detail features, preserving image details while improving 3D pose accuracy.Its single-frame network is reported to outperform the compared methods.
- Monocular-video pose recovery: Prior monocular-video pose methods use FC networks, LSTMs, or TCNs, but TCN-based approaches lack temporal supervision.The paper introduces adversarial training using temporal order as supervision.
- Monocular-video mesh recovery: Video mesh recovery has optimization-based and CNN-based approaches; optimization-based smoothing is less efficient, while HMR-video models motion dynamics with TCNs.The proposed method is reported to outperform the referenced prior methods on 3DPW and Human3.6M.
3. Method
The framework separates spatial mesh factors with DSD, then models temporal coherence using SATN and shuffled-frame order recovery. It represents bodies with SMPL parameters and supports supervision from heterogeneous 2D/3D data.
- 3.1. 3D Human Body Representation: The method estimates an 85-dimensional SMPL-based body representation from single images and monocular videos.SMPL maps shape and pose parameters to a mesh with 6890 vertices; the representation also includes camera rotation, translation, and scale.
- 3.2. DSD Module: DSD independently extracts skeleton and detailed features, then combines them with a bilinear transformation while preserving their decoupling.The module is lightweight, end-to-end, and pluggable into other pose-estimation networks.
- 3.2. DSD Module: The DSD skeleton branch derives 2D/3D joint coordinates from normalized heatmaps using differential integral regression.For 2D heatmaps, the depth dimension is set to 1.
- 3.3. Self-attention Temporal Network: SATN combines self-attention with TCN to model short- and long-term temporal coherence from DSD-reorganized frame features.Self-attention reduces inter-frame path lengths and expands the effective receptive field available to the TCN.
- 3.3. Self-attention Temporal Network: An unsupervised adversarial strategy shuffles frame features and trains sequence sorting to recover their original temporal order.The recovered order is restored before TCN processing, while the sorting module provides supervision for temporal representation learning.
- 3.4. Loss Functions: The framework optimizes estimated body parameters using available 2D/3D supervision and additional parameter-based losses.This design aims to make fuller use of datasets with limited full 3D annotations.
4. Experiments
Experiments evaluate the method on Human3.6M and 3DPW, including state-of-the-art comparisons, ablations, decoupling effectiveness, and temporal stability. Results show improvements from DSD, SATN, self-attention, and adversarial training, with strong generalization to 3DPW without fine-tuning.
- Experimental setup: Human3.6M provides controlled video training data, while 3DPW evaluates richer activities and generalization without fine-tuning.Human3.6M is the only dataset with 3D annotations used for training.
- State-of-the-art comparisons: The proposed method achieves state-of-the-art results on 3DPW without fine-tuning and outperforms HMR-video by 13.2% PA-MPJPE.It also surpasses HMR-video-L on all evaluated 3DPW splits.
- State-of-the-art comparisons: On Human3.6M, the proposed method outperforms HMR-video and HMR-video-L by 26.6% and 23.8% PA-MPJPE, respectively.The method also greatly outperforms SMPLify on 3DPW.
- DSD ablation: Adding DSD improves Direct by 45.3% MPJPE and 29.4% PA-MPJPE, while reducing HMR errors by 31.6% MPJPE and 23.7% PA-MPJPE.DSD also outperforms STN by 14% MPJPE and 27.9% PA-MPJPE under the same evaluation protocol.
- DSD ablation: Replacing estimated skeletons with random ones changes DSD-predicted body postures coherently, unlike outputs from Concat.The experiment compares replacement of skeleton or detailed features to assess decoupling effectiveness.
- Temporal modeling: SATN adds 4.2%/7.3% improvement in MPJPE/PA-MPJPE on Human3.6M and improves prediction smoothness, while self-attention and adversarial training strengthen temporal modeling.DSD combined with SATN outperforms backbone-based methods by 26.6% MPJPE and 27.5% PA-MPJPE; Concat produces larger MPJVE and MPJAE.
5. Conclusion
The framework recovers 3D human meshes from single images and monocular videos using a skeleton-disentangled representation, with spatial and temporal components designed for efficient modeling.
- The end-to-end framework recovers 3D human meshes from single images and monocular videos through skeleton-disentangled representations.
- Decoupling the skeleton reduces task complexity and 3D pose error.
- The DSD module bridges 2D/3D pose estimation and 3D mesh recovery.
- SATN explores both long- and short-term temporal coherence.
- The method provides an efficient baseline for human motion analysis, 3D virtual try-on, and multi-person 3D mesh recovery.