Source-linked AI summary
End-to-End Human Pose and Mesh Reconstruction with Transformers
Kevin Lin, Lijuan Wang, Zicheng Liu
TL;DR
Single-image human mesh reconstruction must handle complex poses and occlusions. METRO uses a transformer encoder with masked vertex modeling to jointly predict joints and vertices through non-local interactions, achieving state-of-the-art results across human and hand reconstruction benchmarks.
Problem
Single-image 3D human pose and mesh reconstruction is challenging because of complex articulated motion and occlusions.
Method
METRO uses a multi-layer transformer encoder with Masked Vertex Modeling to jointly reconstruct 3D body joints and mesh vertices while learning vertex-vertex and vertex-joint interactions.
Results
METRO advances state-of-the-art performance on Human3.6M, 3DPW, and FreiHAND datasets.
Takeaways & Limitations
METRO generalizes beyond human bodies to 3D hand reconstruction and learns input-dependent interactions without being limited by mesh topology.
Takeaways & Limitations
Masked vertex modeling becomes less effective when more than 30% of input queries are masked because many missing queries make training more difficult.
Abstract
from arXiv · showhide
We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct 3D human pose and mesh vertices from a single image. Our method uses a transformer encoder to jointly model vertex-vertex and vertex-joint interactions, and outputs 3D joint coordinates and mesh vertices simultaneously. Compared to existing techniques that regress pose and shape parameters, METRO does not rely on any parametric mesh models like SMPL, thus it can be easily extended to other objects such as hands. We further relax the mesh topology and allow the transformer self-attention mechanism to freely attend between any two vertices, making it possible to learn non-local relationships among mesh vertices and joints. With the proposed masked vertex modeling, our method is more robust and effective in handling challenging situations like partial occlusions. METRO generates new state-of-the-art results for human mesh reconstruction on the public Human3.6M and 3DPW datasets. Moreover, we demonstrate the generalizability of METRO to 3D hand reconstruction in the wild, outperforming existing state-of-the-art methods on FreiHAND dataset. Code and pre-trained models are available at https://github.com/microsoft/MeshTransformer.
1. Introduction
METRO addresses the difficulty of reconstructing 3D human pose and mesh from one image by modeling global interactions with a transformer. It jointly predicts joints and vertices, achieves strong benchmark results, and extends to 3D hands.
- Single-image 3D human reconstruction is challenging because articulated motion and occlusions complicate pose and mesh estimation.
- Prior non-parametric methods model local vertex neighborhoods or use heatmaps, limiting efficient learning of non-local vertex interactions.
- METRO uses a multi-layer transformer encoder and Masked Vertex Modeling to jointly learn vertex-vertex and vertex-joint interactions.
- The framework discovers short- and long-range interactions that support reconstruction under large pose variations and occlusions, while remaining applicable to different mesh types.
- METRO simultaneously reconstructs 3D body joints and mesh vertices from a single input image.
- METRO achieves new state-of-the-art performance on Human3.6M and 3DPW and reaches first place on the FreiHAND leaderboard at submission.
2. Related Works
Earlier approaches use parametric models or topology-constrained direct representations, while METRO applies transformer self-attention to regress joints and vertices without fixed mesh topology.
- Human mesh reconstruction from a single image is difficult because of complex pose variations, occlusions, and limited 3D training data.
- Parametric approaches estimate pose and shape coefficients using models such as SMPL, STAR, and MANO, often with additional body priors or temporal information.
- Direct-regression methods represent the body with meshes, volumetric spaces, or occupancy fields, choosing output representations for specific applications.
- GCNN-based methods model neighborhood interactions using predefined mesh topology but are less efficient for longer-range interactions.
- Transformer self-attention provides a mechanism for modeling dependencies without regard to their distance, motivating its use for global mesh interactions.
- METRO uses a multi-layer transformer with progressive dimensionality reduction to regress joint and vertex 3D coordinates.
3. Method
METRO uses CNN features and a transformer encoder to predict 3D body joints and mesh vertices in parallel, with masked modeling and progressive dimensionality reduction supporting joint–vertex interactions.
- Architecture: The transformer encoder outputs 3D body-joint and mesh-vertex coordinates simultaneously from image features and joint or vertex queries.Queries incorporate template-mesh 3D coordinates as positional information.
- Architecture: The framework extracts a 2048-dimensional image feature vector with a CNN before transformer-based 3D regression.The CNN is pretrained on ImageNet, and the resulting feature vector is input to the transformer.
- Architecture: Progressive dimensionality reduction adapts the transformer architecture to predicting 3D coordinates rather than maintaining constant hidden dimensionality.Linear projections reduce hidden-embedding dimensionality after transformer layers.
- Masked Vertex Modeling: Masked Vertex Modeling randomly masks input queries while requiring regression of all joints and vertices, activating bidirectional attention across relevant body elements.The objective simulates partial occlusions and supports short- and long-range interactions without relying on mesh distance or topology.
- Training: The method projects predicted 3D joints into 2D using learned camera parameters and minimizes reprojection errors against 2D joint ground truth.The camera parameters are learned with a linear layer on transformer outputs.
- Training: Training mixes datasets with and without paired image-mesh annotations, using sample-specific flags to select available 3D and 2D supervision.The objective can incorporate 3D vertices, 3D joints, regressed joints, and 2D reprojection losses.
- Mesh Resolution: Due to hardware memory constraints, the transformer processes a 431-vertex coarse mesh that learnable MLPs upsample to the original 6890-vertex SMPL topology.The transformer and MLPs are trained end-to-end, while the method can process arbitrary mesh sizes in principle.
4. Experimental Results
METRO outperforms prior methods on Human3.6M and 3DPW, while ablations and attention analyses examine masked vertex modeling, non-local interactions, and backbone choices. It also generalizes to in-the-wild 3D hand reconstruction on FreiHAND.
- Main Results: METRO outperforms prior state-of-the-art methods on both 3DPW and Human3.6M.On 3DPW, it outperforms the video-based VIBE despite using only an image; on Human3.6M, gains are especially notable for PA-MPJPE.
- Main Results: METRO performs well across datasets with severe occlusions and differing body-shape estimation challenges.The authors interpret its results on 3DPW and Human3.6M as demonstrating robustness to occlusions and accurate body-shape regression.
- Ablation Study: Masked Vertex Modeling significantly improves Human3.6M reconstruction performance, with benefits becoming less prominent beyond 30% masked queries.The authors attribute this saturation to the greater training difficulty caused by large numbers of missing queries.
- Non-local Interactions: Attention visualizations show input-dependent, non-local interactions between body joints and mesh vertices, including attention to visible body parts during occlusion.The analyses cover 14 body joints and 431 mesh vertices and visualize attention weights across queries and predicted outputs.
- Input Representations: Increasing the number of filters in HRNet’s high-resolution feature maps improves mesh regression performance on Human3.6M.The transformer remains competitive with a ResNet50 backbone, while larger HRNet high-resolution channels provide further improvement.
- Generalization to 3D Hand: METRO outperforms prior methods by a large margin on FreiHAND, demonstrating generalization to in-the-wild 3D hand reconstruction.The model is trained on FreiHAND from scratch and achieved first place on its leaderboard at the time of submission without external training data.
5. Conclusion
METRO is a mesh-transformer framework that learns non-local interactions among body joints and mesh vertices for single-image reconstruction. It advances state-of-the-art performance across 3DPW, Human3.6M, and FreiHAND, while generalizing to 3D hand reconstruction.
- Masked Vertex Modeling learns non-local interactions among body joints and mesh vertices.
- METRO advances state-of-the-art performance on 3DPW, Human3.6M, and FreiHAND datasets.
- The approach generalizes to 3D hand reconstruction and uses predictions based on important joints and vertices regardless of mesh topology.
APPENDICES
The appendices add comparisons with additional methods and discuss transformer research in related 3D vision tasks. On Human3.6M, METRO is reported to outperform the added prior works by a large margin.
- Figure 6 adds HKMR, Zeng et al., Zhang et al., DenseRaC, Arnab et al., and HoloPose to comparisons on 3DPW and Human3.6M.
- The additional methods include SMPL-parameter regression, hierarchical kinematic priors, rotation-angle guidance, continuous UV maps, and UV-map inpainting.
- METRO outperforms all added prior works by a large margin on their reported Human3.6M results.
- Related work is exploring transformers for multi-view human pose estimation and hand pose estimation based on point clouds.
B. Implementation Details and Computation Resource
The implementation uses a multi-GPU PyTorch and Huggingface training setup, with masked vertex modeling implemented through a special token. Ablations compare dimensionality-reduction and positional-encoding schemes, while runtime and parameter details are tabulated.
- Training uses 8 NVIDIA V100 GPUs, batch size 32, Adam optimization, and 200 epochs over approximately 5 days.
- Inference runs at about 12 fps with batch size 1 on a single NVIDIA P100 GPU and around 24 fps with batch size 32.
- Masked vertex modeling replaces randomly selected input queries with a predefined 2051-D [MASK] token.
- Tables report parameter counts and inference time per image, with runtime estimated using batch size 1.
- The (H+3)→3 and (H+3)→H/2→3 dimensionality-reduction baselines produce poor results in the ablation study.
- Sinusoidal positional encoding performs slightly worse than the method’s coordinate-based encoding.
E. Qualitative Results
Qualitative results show that METRO reconstructs plausible human meshes in severe bending, occlusion, and sitting scenarios. Without architectural modifications, it also produces effective and occlusion-robust 3D hand reconstructions.
- METRO reconstructs a reasonable human mesh with correct pose when the subject is heavily bending.
- In vehicle-occlusion scenes, METRO performs more robustly than prior image-based methods that fail to generate correct human meshes.
- METRO reconstructs a better human mesh than previous state-of-the-art methods when the subject is sitting on a chair.
- Without modifying the network architecture, METRO works well for 3D hands and remains robust to occlusions.
F. Non-local Interactions of Hand Joints
METRO’s learned self-attention captures non-local relationships among hand joints. Analysis on FreiHAND visualizes these interactions across all 21 joints and identifies a strong wrist–ring-finger connection.
- The attention map contains 21 rows and 21 columns, representing attention between the 21 hand joints.Each pixel (i, j) measures how strongly joint i attends to joint j.
- Darker pixels indicate stronger self-attention between the corresponding hand joints.
- The wrist receives strong attention from joints on the ring finger.The authors attribute this pattern to greater motion correlation between the pinky and ring finger than between the pinky and other fingers.
G. Test-Time Augmentation for FreiHAND
The FreiHAND experiments evaluate test-time augmentation by averaging predictions from transformed versions of each test image after 3D alignment.
- Test-time augmentation applies different rotations and scaling transformations to each FreiHAND test image.
- The transformed images are fed into METRO, and their predicted output meshes are averaged.
- Procrustes analysis aligns the output meshes before averaging them in 3D.
H. Limitations
METRO’s performance is constrained when test images differ substantially from its training data. Reported failures involve atypical body shapes, unseen gloves, and applying a right-hand model to left-hand images.
- METRO may fail when the testing sample is very different from the training data.The paper illustrates such failures with test images downloaded from the Internet.
- The method may not faithfully reconstruct muscles when the target body shape differs from SMPL-style training data.
- The model can produce an incorrect mesh when glove data are absent from the training set.
- Applying a pretrained right-hand model to left-hand images does not work well because the method is mesh-specific.The paper identifies unified modeling for different 3D objects as future work.