Source-linked AI summary
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction
Nikos Kolotouros, Georgios Pavlakos, Kostas Daniilidis
TL;DR
Single-image 3D human pose and shape estimation is difficult, and direct regression of SMPL parameters is limited by the parameter space. The paper instead uses a Graph-CNN to regress mesh vertices, achieving stronger results than direct-parameter baselines and state-of-the-art performance among model-based approaches.
Problem
Single-image 3D human pose and shape estimation is challenging, while direct SMPL parameter regression is constrained by the model space and difficult rotation representations.
Method
A Graph-CNN attaches image features to a template human mesh and directly regresses the 3D locations of its vertices.
Results
The method outperforms direct model-parameter baselines across input representations and achieves state-of-the-art results among model-based pose estimation approaches.
Takeaways & Limitations
The recovered mesh can optionally be used to regress parameters for a specific body model, while retaining a non-parametric mesh representation.
Takeaways & Limitations
Typical failures occur with challenging poses, severe self-occlusions, and interactions among multiple people.
Abstract
from arXiv · showhide
This paper addresses the problem of 3D human pose and shape estimation from a single image. Previous approaches consider a parametric model of the human body, SMPL, and attempt to regress the model parameters that give rise to a mesh consistent with image evidence. This parameter regression has been a very challenging task, with model-based approaches underperforming compared to nonparametric solutions in terms of pose estimation. In our work, we propose to relax this heavy reliance on the model's parameter space. We still retain the topology of the SMPL template mesh, but instead of predicting model parameters, we directly regress the 3D location of the mesh vertices. This is a heavy task for a typical network, but our key insight is that the regression becomes significantly easier using a Graph-CNN. This architecture allows us to explicitly encode the template mesh structure within the network and leverage the spatial locality the mesh has to offer. Image-based features are attached to the mesh vertices and the Graph-CNN is responsible to process them on the mesh structure, while the regression target for each vertex is its 3D location. Having recovered the complete 3D geometry of the mesh, if we still require a specific model parametrization, this can be reliably regressed from the vertices locations. We demonstrate the flexibility and the effectiveness of our proposed graph-based mesh regression by attaching different types of features on the mesh vertices. In all cases, we outperform the comparable baselines relying on model parameter regression, while we also achieve state-of-the-art results among model-based pose estimation approaches.
1. Introduction
Monocular 3D human pose and shape reconstruction is difficult because of ambiguity, limited training data, and imaging variation. The paper addresses limitations of direct SMPL parameter regression by using Graph-CNN mesh regression.
- Monocular reconstruction is challenging because of excessive ambiguity, limited training data, and varied imaging conditions.
- Direct regression of SMPL parameters is constrained by the model's limited coverage and difficult rotation representations.SMPL does not model hand pose or facial expressions, while rotation representations can introduce periodicity, non-minimality, or discontinuities.
- The proposed Graph-CNN directly regresses 3D mesh vertices while explicitly encoding template-mesh structure and spatial locality.Image features are attached to template vertices, and graph processing predicts each vertex's 3D location.
- The framework supports RGB, semantic segmentation, and dense-correspondence features, outperforming corresponding direct-parameter baselines.Its graph-based processing is largely agnostic to the input representation.
- The method reformulates human pose and shape estimation as mesh-vertex regression rather than direct model-parameter regression.The mesh topology is retained while the parametric space is no longer the direct regression target.
- The approach achieves state-of-the-art results among model-based pose estimation approaches.
2. Related work
Prior work includes optimization-based, nonparametric, and graph-based approaches for recovering human pose and shape. This paper distinguishes itself by using a Graph-CNN for single-image mesh reconstruction rather than learning a generative shape model from 3D shapes.
- Optimization-based shape estimation: Earlier full-body pose and shape methods primarily used optimization-based recovery of parameters for parametric models.
- Related mesh regression: The proposed approach differs from prior related methods by regressing mesh-vertex locations with a Graph-CNN and accepting multiple image-derived input representations.
- Nonparametric shape estimation: Nonparametric voxel methods reconstruct voxelized body sculptures but have a stated main drawback beyond that reconstruction.
- Graph CNNs: Unlike prior Graph-CNN work on generative shape models or object meshes, this method performs single-image human shape reconstruction from image input.The paper notes that Graph CNNs themselves are not new; its contribution is applying them to this reconstruction setting.
3. Technical approach
The approach regresses 3D mesh vertices with a Graph CNN, then optionally recovers SMPL parameters from the predicted shape. It combines image-derived features, mesh-structured processing, camera estimation, and shape and keypoint supervision.
- Image-based feature extraction: A ResNet-50 feature extractor produces a 2048-D representation from RGB images, part segmentation, or DensePose input.The final fully connected layer is removed, retaining the feature vector after average pooling.
- Graph-based mesh regression: The Graph CNN attaches image features to template-mesh vertices and regresses each vertex's 3D coordinates.The template mesh has N vertices, and the graph architecture explicitly processes features over its structure.
- Graph-based mesh regression: Graph convolution applies per-vertex transformations followed by neighborhood averaging, encouraging similar neighboring features and smooth output shapes.This design removes the observed need for an additional shape smoothness loss.
- Camera and joint projection: The network also predicts weak-perspective camera scale and translation from the graph embedding, while 3D joints are obtained from predicted vertices and projected to the image plane.The projection uses the predicted 3D joints, scale s, and 2D translation t.
- Training supervision: Training combines per-vertex shape and projected-keypoint L1 losses, with weaker supervision using only keypoint loss when full 3D shape ground truth is unavailable.The authors report that L1 loss trains more stably and performs better than L2 loss.
- SMPL parameter recovery: A separate MLP regresses SMPL pose and shape parameters from the predicted 3D shape, using a two-step procedure that first fixes the non-parametric shape network.End-to-end training was observed to reduce both parametric and non-parametric shape performance.
4. Empirical evaluation
The empirical evaluation compares graph-based mesh regression with direct SMPL parameter regression and fully connected mesh regression across datasets, inputs, and evaluation settings. The proposed approach generally outperforms relevant baselines, while failures concentrate in difficult multi-person or occluded scenarios.
- Datasets: The evaluation uses Human3.6M and UP-3D for training, and Human3.6M and LSP for evaluation.Human3.6M results use protocols P1 and P2 with MPJPE and Reconstruction error; LSP provides a 2D pose test set with segmentation annotations.
- Regression target: Graph-based mesh regression outperforms direct SMPL parameter regression and the strong HMR baseline on Human3.6M.The comparison includes mesh regression with and without subsequent SMPL parameter regression.
- Graph architecture: Graph-CNN mesh regression outperforms naive fully connected mesh regression by leveraging the mesh structure during vertex-coordinate prediction.The fully connected alternative produces non-smooth meshes because it cannot directly exploit graph topology.
- Input representation: The method is evaluated with different input representations, including RGB images, semantic part segmentation, and dense correspondences.The framework is described as largely agnostic to the type of features attached to mesh vertices.
- Benchmark results: The approach outperforms previous baselines on Human3.6M and is competitive with the state of the art on LSP segmentation evaluation.Table 4 reports lower Reconstruction errors than previous baselines, while Table 5 reports accuracies and F1 scores.
- Efficiency and qualitative analysis: 33ms inference time on a 2080 Ti GPU makes single-image network inference effectively real-time.Qualitative examples show both nonparametric meshes and corresponding SMPL meshes; typical failures involve challenging poses, severe self-occlusions, or multiple people.
5. Summary
The paper reduces reliance on direct SMPL parameter regression by predicting mesh vertices with a Graph-CNN while retaining the SMPL template topology. This hybrid representation improves model-based pose estimation and supports later recovery of SMPL parameters from predicted geometry.
- Problem and scope: The paper targets single-image human pose and shape estimation while reducing reliance on a parametric body model.It retains the SMPL mesh topology but changes the prediction target from model parameters to 3D mesh-vertex locations.
- Approach: The Graph-CNN explicitly encodes mesh structure and processes image features attached to mesh vertices.This architecture supports convolutional mesh regression of the 3D vertex locations.
- Results: The approach outperforms direct model-parameter regression across varied input representations and achieves state-of-the-art results among model-based pose-estimation methods.The conclusion states this pattern without restricting it to a single dataset or representation.
- Implications: The nonparametric representation may support recovering body aspects absent from many human models, including hand articulation, facial expressions, clothing, and hair.These are presented as future opportunities rather than demonstrated capabilities of the current system.