PaperlayerView exact revision on arXiv

Source-linked AI summary

I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image

Gyeongsik Moon, Kyoung Mu Lee

arXiv:2008.03713v2cs.CV

TL;DR

Prior image-based methods often directly regress human-mesh-model parameters, breaking pixel spatial relationships and failing to model prediction uncertainty. I2L-MeshNet instead predicts per-lixel likelihoods on 1D heatmaps for mesh-vertex coordinates, with a memory-efficient representation and cascaded pose-to-mesh architecture. It outperforms previous methods across multiple 3D human pose and mesh benchmarks, while evaluation uses pseudo-groundtruth or imperfectly aligned SMPL parameters in supported settings.

  • Problem

    Directly regressing mesh-model parameters breaks spatial relationships among image pixels and cannot model prediction uncertainty, making training harder.

  • Method

    I2L-MeshNet predicts per-lixel likelihoods on 1D heatmaps for each mesh-vertex coordinate, using a cascaded PoseNet and MeshNet architecture.

  • Results

    I2L-MeshNet outperforms previous 3D human pose and mesh estimation methods across various benchmark datasets.

  • Takeaways & Limitations

    Lixel-based 1D heatmaps preserve image spatial relationships, model prediction uncertainty, and use less GPU memory at the same resolution than voxel-based 3D heatmaps.

  • Takeaways & Limitations

    The method may benefit from more accurately aligned in-the-wild image–3D mesh data, and SMPL parameters obtained with SMPLify-X are not perfectly aligned to Human3.6M 3D joints.

Abstract

from arXiv · show

Most of the previous image-based 3D human pose and mesh estimation methods estimate parameters of the human mesh model from an input image. However, directly regressing the parameters from the input image is a highly non-linear mapping because it breaks the spatial relationship between pixels in the input image. In addition, it cannot model the prediction uncertainty, which can make training harder. To resolve the above issues, we propose I2L-MeshNet, an image-to-lixel (line+pixel) prediction network. The proposed I2L-MeshNet predicts the per-lixel likelihood on 1D heatmaps for each mesh vertex coordinate instead of directly regressing the parameters. Our lixel-based 1D heatmap preserves the spatial relationship in the input image and models the prediction uncertainty. We demonstrate the benefit of the image-to-lixel prediction and show that the proposed I2L-MeshNet outperforms previous methods. The code is publicly available https://github.com/mks0601/I2L-MeshNet_RELEASE.

1 Introduction

3D human pose and mesh estimation must recover joint and mesh-vertex locations from a single image despite articulation and 2D-to-3D ambiguity. I2L-MeshNet addresses limitations of direct parameter regression with lixel-based 1D heatmaps that preserve image spatial relationships and model uncertainty.

  • Task: 3D human pose and mesh estimation simultaneously recovers 3D semantic joint and mesh-vertex locations, despite complicated articulation and 2D-to-3D ambiguity.The task supports applications including virtual/augmented reality and human action recognition.
  • Prior approaches: Most prior methods regress SMPL or MANO parameters from an image, while relatively few model-free methods estimate mesh-vertex coordinates directly.SMPL and MANO represent human poses and identities through pose and identity parameters.
  • Motivation: Direct target representations break pixel spatial relationships and cannot model prediction uncertainty, making training harder and potentially reducing test accuracy.Heatmap-based pose methods address these representation limitations, but conventional 3D heatmaps become infeasible for dense meshes.
  • Proposed method: I2L-MeshNet predicts per-lixel likelihoods on 1D heatmaps for each mesh-vertex coordinate, extending heatmap-based 3D pose estimation to pose and mesh estimation.A lixel is defined as a quantized cell in one-dimensional space; the method uses a model-free representation.
  • Proposed method: Lixel-based 1D heatmaps have memory complexity linear in heatmap resolution, enabling sufficient resolution for dense mesh-vertex localization.This avoids the computational infeasibility of predicting 3D heatmaps for thousands of mesh vertices.
  • Proposed method: The cascaded architecture uses PoseNet to estimate joint heatmaps, then MeshNet combines PoseNet output with image features to estimate mesh-vertex heatmaps.Joint locations provide coarse information about mesh-vertex locations and can substantially increase estimation accuracy.
  • Results: I2L-MeshNet outperforms previous 3D human pose and mesh estimation methods across multiple benchmark datasets.The paper presents qualitative body and hand mesh results on MSCOCO and FreiHAND.

3 I2L-MeshNet

I2L-MeshNet uses a cascaded PoseNet–MeshNet architecture to predict lixel-based 1D heatmaps for human joints and mesh vertices, then converts them into continuous 3D coordinates and final pose and mesh outputs.

  • PoseNet: PoseNet predicts three lixel-based 1D heatmaps for every human joint from the input image.The x- and y-axis heatmaps represent image-space coordinates, while the z-axis heatmap represents root-joint-relative depth.
  • PoseNet: PoseNet converts discretized heatmaps into continuous joint coordinates using soft-argmax.The depth discretization size is D = 8h = 8w.
  • MeshNet: MeshNet combines PoseNet-derived image features and 3D Gaussian joint heatmaps to predict lixel-based 1D heatmaps for all mesh vertices.The joint heatmaps provide coarse but important information for mesh estimation, and MeshNet predicts vertex heatmaps in the same manner as PoseNet.
  • Final 3D human pose and mesh: MeshNet converts vertex heatmaps into continuous mesh coordinates, which are transformed into final 3D mesh and pose outputs.The final outputs use camera back-projection, inverse affine transformation, root-joint depth offset, and the SMPL or MANO joint regression matrix.
  • Loss functions: I2L-MeshNet is trained end-to-end with pose, mesh vertex, mesh normal, and mesh edge-length losses.The z-axis loss and mesh-related losses become zero when the corresponding ground-truth supervision is unavailable; λ = 0.1 weights Lnormal, and gradients are not backpropagated before PHG.

4 Implementation details

The implementation uses a ResNet-50 backbone with standard optimization and image-augmentation procedures, resizing cropped human images to 256×256 pixels.

  • Model and optimization: The backbone is initialized from ImageNet-pretrained ResNet-50, while remaining weights use a Gaussian initialization with σ = 0.001.Training uses PyTorch and the Adam optimizer with mini-batches of 48.
  • Input processing: Human regions are cropped using ground-truth bounding boxes during training and testing, or Mask R-CNN when test-time boxes are unavailable.The cropped image is resized to 256×256, giving D = 64 and h = w = 8.
  • Data augmentation: Training augmentation includes scaling by ±25%, rotation by ±60°, random horizontal flipping, and color jittering by ±20%.The initial learning rate is 10^-4 and is reduced during training.

5 Experiment

Experiments evaluate target representations, heatmap designs, cascading strategies, and comparisons with prior methods across Human3.6M, 3DPW, and FreiHAND. I2L-MeshNet’s heatmap and lixel designs achieve strong accuracy and efficiency, while comparisons depend on training supervision.

  • Ablation study: Human3.6M is used for ablation studies, with MPJPE and PA MPJPE serving as evaluation metrics.The ablation models are trained and tested on this large-scale benchmark.
  • Ablation study: Heatmap-based mesh estimation achieves the lowest errors with the fewest parameters and small GPU memory usage among target representations.The comparison includes SMPL parameters, vertex coordinates, and heatmaps.
  • Ablation study: Modeling prediction uncertainty lowers errors and parameter count relative to direct coordinate regression, while also outperforming SMPL parameter regression.The compared models use the same targets but different representations; heatmaps model uncertainty whereas direct regression does not.
  • Ablation study: Fully convolutional x- and y-axis heatmaps preserve pixel relationships and achieve the best performance with the fewest parameters, while using similar memory to SMPL regression.The spatially preserving final model retains the memory advantage of the compared target representations.
  • Ablation study: Lixel-based heatmaps use much less GPU memory than pixel- and voxel-based heatmaps at the same resolution, with memory complexity linear in D.This enables high-resolution heatmaps needed for dense mesh vertex localization; voxel resolution cannot be enlarged within a 12 GB limit.
  • Ablation study: Using PoseNet output in the cascaded PoseNet–MeshNet architecture improves accuracy over directly predicting mesh heatmaps from the image.The study also evaluates an upper-bound variant that supplies ground-truth 3D pose to MeshNet.
  • Comparison with state-of-the-art methods: When all methods are trained on Human3.6M and MSCOCO, I2L-MeshNet significantly outperforms previous methods by a large margin on Human3.6M and 3DPW.A second comparison using methods trained on different datasets finds that the performance gap is significantly reduced.
  • Comparison with state-of-the-art methods: With identical 3D supervision, I2L-MeshNet is more accurate than model-based methods, whereas model-based methods can match or exceed it using additional in-the-wild 2D pose data.The authors attribute this difference to model-based methods benefiting from weak supervision unavailable to model-free approaches.

6 Conclusion

I2L-MeshNet estimates 3D human pose and mesh from a single RGB image using lixel-based 1D heatmaps that preserve spatial relationships and model prediction uncertainty. It uses less GPU memory than voxel-based heatmaps while achieving better accuracy and outperforms previous methods across datasets.

  • I2L-MeshNet estimates 3D human pose and mesh from a single RGB image using lixel-based 1D heatmaps.
  • The lixel-based representation preserves spatial relationships in the input image and models prediction uncertainty.
  • Lixel-based 1D heatmaps require much less GPU memory than voxel-based 3D heatmaps at the same resolution while producing better accuracy.
  • I2L-MeshNet outperforms previous 3D human pose and mesh estimation methods on various datasets.

7 Various settings of I2L-MeshNet

The experiments examine where marginalization occurs and how it is performed in I2L-MeshNet. Early marginalization reduces GPU memory but increases errors, while average pooling achieves the lowest errors among the compared marginalization methods.

  • Marginalization location: Early marginalization increases MPJPE and PA MPJPE while requiring less GPU memory.The comparison uses marginalization at the ResNet output rather than after the final upsampling module.
  • Marginalization location: Early marginalization can create spatial ambiguity because marginalized 1D feature maps are generated from multiple 2D feature maps.
  • Experimental setting: The marginalization experiments are conducted on Human3.6M, with the z-axis heatmap prediction unchanged in the early-marginalization setting.
  • Marginalization method: Average pooling achieves the lowest MPJPE and PA MPJPE among the compared marginalization methods.
  • Marginalization method: Weighted-sum marginalization has lower error than max pooling but higher error than average pooling.The weighted-sum implementation uses large convolutional kernels for x- and y-axis lixel heatmap prediction.

8 Comparison with previous 2.5D heatmap regression

Compared with a 2.5D heatmap representation, I2L-MeshNet’s lixel-based 1D heatmap achieves lower MPJPE and uses less GPU memory under the same resolution.

  • The lixel-based 1D heatmap achieves significantly lower MPJPE than the 2.5D heatmap under the same resolution.
  • The lixel-based 1D heatmap requires much less GPU memory than the compared 2.5D heatmap.
  • The 2.5D heatmap combines an xy pixel-based 2D heatmap with a z heatmap containing root-joint-relative depths at activated xy positions.
  • Unlike the lixel-based representation, the compared 2.5D heatmap predicts z-axis depth values rather than z-axis likelihoods and therefore cannot model z-axis prediction uncertainty.

9 Effect of each loss function

The loss-function experiments evaluate pose supervision and mesh regularization. Additional pose loss improves human joint-aligned mesh prediction, while normal-vector and edge-length losses improve visual quality with marginal effects on pose metrics.

  • Pose loss: Additional mesh pose loss improves human joint-aligned mesh prediction.Both compared models are trained on Human3.6M.
  • Mesh regularization losses: Normal-vector and edge-length losses improve the visual quality of output meshes.The comparison is shown in Figure 4.
  • Mesh regularization losses: Normal-vector and edge-length losses marginally affect MPJPE and PA MPJPE.The corresponding models are trained on Human3.6M and MSCOCO.

10 Accuracy of PoseNet

PoseNet is evaluated using MPJPE and PA MPJPE across datasets, with errors computed against SMPL or MANO ground-truth 3D joint coordinates.

  • PoseNet is trained jointly with MeshNet by minimizing the loss function L.
  • MPJPE and PA MPJPE are reported for PoseNet on each dataset.
  • FreiHAND MPJPE is unavailable because its official evaluation server does not support that metric.

11 Pseudo-groundtruth SMPL parameters of Human3.6M dataset

Because Mosh-based SMPL parameter distributions are unavailable under licensing and source-code constraints, the paper uses SMPLify-X fits from Human3.6M ground-truth 3D joints instead.

  • Earlier works used SMPL parameters obtained by applying Mosh to Human3.6M marker data as ground-truth parameters.
  • Mosh parameter distributions are currently disallowed by licensing, and Mosh source code is not publicly released.
  • The paper alternatively obtains Human3.6M ground-truth SMPL parameters by applying SMPLify-X to ground-truth 3D joint coordinates.
  • Table 13 compares SMPLify-X fitting MPJPE with state-of-the-art 3D human pose estimation methods.
  • Table 14 compares state-of-the-art methods and I2L-MeshNet using MPVPE and MPJPE on SURREAL.

12 Evaluation on SURREAL

On SURREAL, I2L-MeshNet is evaluated with root-aligned MPVPE and MPJPE and qualitatively compared with GraphCMR, producing visually more pleasant meshes.

  • 12 Evaluation on SURREAL: SURREAL contains 67K clips synthesized by animating the SMPL body model.
  • 12 Evaluation on SURREAL: Evaluation uses root-aligned MPVPE and MPJPE, with MPVPE measuring averaged per-vertex Euclidean mesh-coordinate error in millimeters.
  • 12 Evaluation on SURREAL: Compared with GraphCMR, I2L-MeshNet produces much more visually pleasant mesh results in qualitative comparisons.
  • 12 Evaluation on SURREAL: The paper attributes GraphCMR’s smoothing tendency to GraphCNN averaging vertex features with neighboring vertices.
Loading 2008.03713v2…