Convolutional Mesh Regression for Single-Image Human Shape Reconstruction

• Monocular reconstruction of complete human 3D pose and shape is challenging because of excessive ambiguity, limited training data, and varied imaging conditions.The paper distinguishes this task from 2D pose or simplistic 3D-skeleton estimation.Source: §1 · p. 1
• SMPL’s 3D-rotation pose parameters can be difficult to regress because of periodicity, non-minimal representations, or discontinuities, and model-based methods underperform 3D-joint regression on pose metrics [23] [26] [3] [24] [29] [38].Earlier learning-based approaches varied their input representation but typically used SMPL parameters as regression targets [15] [18] [28] [31] [43].Source: §1 · pp. 1–2
• The method retains the SMPL template topology but regresses each vertex’s 3D location, processing image features attached to template vertices with a Graph-CNN that encodes mesh structure and spatial locality.A typical CNN extracts image features, which are attached to the template vertices before graph-based processing.Source: §1 · p. 2, Figure 2 · p. 2
• After recovering the mesh geometry without committing to a parametric space, the method can reliably regress parameters for a specific body model when needed.This optional step makes the vertex-regression approach comparable to model-based methods.Source: Figure 1 · p. 1, §1 · p. 2
• Across RGB pixels [15], semantic part segmentation [28], and dense correspondences, the method outperforms direct-parameter baselines and achieves state-of-the-art results among model-based pose-estimation approaches.The paper reports this flexibility and comparison across different input feature types.Source: §1 · p. 2

Optimization-based shape recovery

• Optimization-based full-shape recovery fits parametric meshes to image evidence, but its non-convex optimization is initialization-sensitive, can reach local minima, and may take several minutes.Examples include SCAPE fitting from annotated landmarks, SMPLify [1], and fitting with surface landmarks, silhouettes, semantic parts, or multiple people [18].Source: §2 · p. 3

Direct parametric regression

• SMPL parameter regression has been widely used, but 3D rotation parameters create regression difficulties and these methods typically underperform skeleton-only pose-estimation baselines.The paper instead regresses mesh-vertex locations with a Graph-CNN and can use input representations such as part-based features, raw-image features, or dense correspondences.Source: §2 · p. 3

Nonparametric shape estimation

• Nonparametric voxel reconstructions offer body geometry but little semantic information, often requiring expensive model fitting for pose; retaining SMPL topology provides dense semantic correspondences and enables parameter regression from vertices [44].The paper contrasts its topology-preserving vertex prediction with voxel-based approaches [44] [12].Source: §2 · p. 3

Graph CNNs

• Prior Graph CNN work includes object-mesh reconstruction and human-shape analysis, completion, and face modeling, whereas this method takes an image—not a 3D shape—as input for human reconstruction.Its contribution is using graph structure and spatial locality to avoid problematic SMPL parameter regression while preserving semantic correspondences [24] [31].Source: §2 · p. 3

3.1. Image-based CNN

• A ResNet-50 image encoder supplies 2048-D features from RGB, part segmentation, or DensePose inputs, while the study focuses on graph processing rather than input representation.For segmentation and DensePose, the authors assume another network provides the input representation.Source: §3.1 · p. 4

3.2. Graph CNN

• The Graph CNN attaches image features to template-mesh vertices and uses graph convolutions to regress each vertex’s 3D coordinates.Neighborhood averaging encourages similar features at adjacent vertices, producing smooth shapes without a separate smoothness loss.Source: §3.2 · p. 4
• The network also predicts weak-perspective scale and translation from the graph embedding, and trains with vertex-shape and projected-joint losses.When only 2D keypoints are available, training omits the shape-loss term and uses the keypoint loss alone, following Omran et al. [28].Source: §3.2 · pp. 4–5

3.3. SMPL from regressed shape

• An MLP regresses SMPL pose and shape parameters from the predicted 3D mesh; rotation-matrix outputs with differentiable SVD projection converge faster than axis-angle outputs.The rotation representation did not improve quantitative results, but was selected as a more practical training option.Source: §3.3 · p. 5

3.4. Implementation details

• The mesh is subsampled by a factor of four and upsampled to reduce redundancy and memory use; two-stage training outperformed end-to-end training for both shapes.The authors train the non-parametric shape network first, then freeze it while training the SMPL-parameter MLP.Source: §3.4 · p. 5

4.1. Datasets

• Training uses Human3.6M and UP-3D [18], with evaluation on Human3.6M and the LSP test set [13].Human3.6M testing uses subjects S9 and S11 and reports Protocols 1 and 2 with MPJPE and Reconstruction error; UP-3D supplies training data, and LSP evaluation uses its test set.Source: §4.1 · pp. 5–6

4.2. Training details

• The image encoder is an ImageNet-pretrained ResNet50 [7], while the Graph CNN and SMPL-parameter MLP are trained from scratch.Training uses Adam, batch size 16, and learning rate 3e – 4 without decay; Human3.6M-only training lasts 10 epochs and mixed Human3.6M/UP-3D training lasts 25.Source: §4.2 · p. 6

4.3. Experimental analysis

• Graph-CNN mesh regression outperforms fully connected vertex regression, while FC regression produces non-smooth meshes because it cannot leverage mesh topology.The paper attributes the graph-based model’s advantage to processing the mesh structure directly.Source: §4.3 · p. 6, Figure 4 · p. 6
• On Human3.6M, the nonparametric mesh and the SMPL mesh regressed from it have similar performance; on LSP segmentation, the approach is competitive with the state of the art.The LSP comparison includes regression-only approaches and approaches with optimization post-processing.Source: Table 3 · p. 7, Table 5 · p. 7
• On Human3.6M Protocol 2, the approach outperforms previous baselines in reconstruction error, and both graph mesh-regression variants outperform the direct SMPL-regression baseline [15].The variants are mesh regression alone and mesh regression followed by SMPL parameter regression.Source: §4.3 · p. 6, Table 1 · p. 6, Table 4 · p. 7
• Typical reconstruction failures involve challenging poses, severe self-occlusions, or interactions among multiple people.The examples include both the predicted nonparametric mesh and the SMPL mesh regressed from it.Source: Figure 5 · p. 7, §4.3 · p. 8, Figure 6 · p. 8
• On a 2080 Ti GPU, single-image network inference takes 33ms, which the paper describes as effectively real-time.Source: §4.3 · p. 8

Summary

• The method retains SMPL topology but first estimates 3D mesh-vertex locations instead of directly predicting model parameters from an image.A Graph-CNN explicitly encodes the mesh structure and processes image features attached to its vertices.Source: §5 · p. 8
• Convolutional mesh regression outperforms relevant direct model-parameter-regression baselines across input representations and achieves state-of-the-art results among model-based pose-estimation approaches.Source: §5 · p. 8
• Future work could address the output mesh’s low resolution and missing shape details, or use the non-parametric representation to capture hand articulation, facial expressions, clothing, and hair.Source: §5 · p. 8