PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Human Mesh Estimation from Virtual Markers

Xiaoxuan Ma, Jiajun Su, Chunyu Wang, Wentao Zhu, Yizhou Wang

arXiv:2303.11726v4cs.CV

TL;DR

Skeleton-based mesh estimators lose body-shape information, while physical markers are not applicable to wild images without markers. The paper learns virtual markers from mocap data and uses them to reconstruct meshes; its approach outperforms state-of-the-art methods on three datasets, especially SURREAL. Its conclusion is that this representation handles diverse body shapes more accurately and efficiently than prevailing skeleton representations.

  • Problem

    Skeleton-based mesh estimation loses body-shape information, while physical markers that capture surface motion cannot be used on wild images without markers.

    Source: §1 · pp. 1–2

  • Method

    The method learns surface-based virtual markers from mocap data, detects their 3D positions in images, and interpolates them into an intact mesh.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The approach outperforms state-of-the-art methods on three datasets and achieves a significant gain on SURREAL, which has varied body shapes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Virtual markers provide a more expressive and accessible intermediate representation than skeletons or physical markers, enabling more accurate and efficient mesh reconstruction, especially for diverse body shapes.

    Source: §5 · p. 8

    Most errors stem from inaccurate 3D virtual-marker estimates; more powerful estimators or more diverse training data may address them.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Inspired by the success of volumetric 3D pose estimation, some recent human mesh estimators propose to estimate 3D skeletons as intermediate representations, from which, the dense 3D meshes are regressed by exploiting the mesh topology. However, body shape information is lost in extracting skeletons, leading to mediocre performance. The advanced motion capture systems solve the problem by placing dense physical markers on the body surface, which allows to extract realistic meshes from their non-rigid motions. However, they cannot be applied to wild images without markers. In this work, we present an intermediate representation, named virtual markers, which learns 64 landmark keypoints on the body surface based on the large-scale mocap data in a generative style, mimicking the effects of physical markers. The virtual markers can be accurately detected from wild images and can reconstruct the intact meshes with realistic shapes by simple interpolation. Our approach outperforms the state-of-the-art methods on three datasets. In particular, it surpasses the existing methods by a notable margin on the SURREAL dataset, which has diverse body shapes. Code is available at https://github.com/ShirleyMaxx/VirtualMarker

1. Introduction

Skeleton-based mesh estimation is efficient but loses body-shape information, while physical markers are restricted to laboratory settings. This work learns virtual markers that can be detected in images and used to reconstruct meshes, outperforming state-of-the-art methods on three datasets.

  • Directly regressing mesh-model parameters is difficult because image-to-parameter mapping is highly nonlinear, image-model alignment is imperfect, and mesh datasets are small.Source: §1 · pp. 1–2
  • Skeleton proxies enable efficient mesh regression and leverage 2D pose data, but discard body-shape information, limiting estimates of lean or obese shapes.Source: §1 · p. 2
  • Physical markers capture subtle surface motions for accurate meshes, but their use is limited to laboratory environments.Source: §1 · p. 2
  • The approach learns virtual markers from mocap data using archetypal analysis, then detects their 3D positions and interpolates the intact mesh.A light network adjusts interpolation weights according to marker prediction confidence.Source: §1 · p. 2
  • The method consistently outperforms state-of-the-art methods on three benchmark datasets, with a significant gain on SURREAL, which has varied body shapes.Source: §1 · p. 2

2. Related work

Earlier mesh estimators optimized model fits or learned image-to-mesh mappings, while proxy representations sought to ease learning. Virtual markers aim to retain more shape information than skeletons while remaining detectable in wild images.

2.1. Optimization-based mesh estimation

  • Optimization-based methods fit mesh parameters to observations, but are sensitive to initialization and can become trapped in local optima.Source: §2.1 · p. 2

2.2. Learning-based mesh estimation

  • Learning-based SMPL regression faces nonlinear image-to-parameter mappings and mesh-image misalignment, making accurate, generalizable models difficult.Source: §2.2 · p. 2
  • Proxy representations include physical markers, IUV images, segmentation masks, and skeletons; physical markers are accurate but unavailable in marker-free wild images.Source: §2.2 · p. 2
  • Virtual markers are designed to reduce pose-and-shape ambiguity compared with skeletons while remaining applicable to wild images; learned markers approximately outline body shape.Source: §2.2 · p. 3, Figure 2 · p. 3

3. Method

The method learns surface-based virtual markers to represent mesh structure, estimates their 3D positions from an image, and reconstructs the mesh with confidence-adjusted coefficients. It trains the framework end-to-end with marker, confidence, and mesh losses.

3.1. The virtual marker representation

  • On H3.6M, reconstruction errors are small for both original and symmetric marker sets, indicating accurate reconstruction of all vertices.Source: Table 1 · p. 3
  • Archetypal analysis learns virtual markers that linearly reconstruct the mesh while remaining visually distinguishable and preferably on the body surface.The learned convex-combination constraints encourage markers near mesh extreme points; markers are also adjusted for left-right symmetry.Source: §3.1 · pp. 3–4

3.2. Mesh estimation framework

  • The framework estimates 3D marker positions with a volumetric CNN and reconstructs the full mesh by multiplying marker positions by a predicted coefficient matrix.Source: §3.2 · p. 4
  • A 3D heatmap represents per-voxel marker likelihood, and each marker position is estimated as the heatmap’s center of mass.Source: §3.2 · p. 4
  • Because monocular occlusion can make marker positions inaccurate, a fully connected layer uses marker heatmap confidence scores to update reconstruction coefficients.Source: §3.2 · p. 4

3.3. Training

  • End-to-end supervised training combines virtual-marker and confidence losses with mesh losses for vertices, joints, surface normals, and edge lengths.The marker loss compares predicted and ground-truth markers; the confidence loss encourages a high heatmap score at the ground-truth marker location.Source: §3.3 · pp. 4–5

4. Experiments

The experiments evaluate virtual-marker mesh estimation on H3.6M, 3DPW, and SURREAL, with ablations examining representation, marker count, and coefficient-matrix updates. The method outperforms state-of-the-art approaches on all three datasets, while qualitative results show generalization to natural images and localized errors under occlusion.

4.1. Datasets and metrics

  • On H3.6M and 3DPW, evaluation reports MPVE for meshes and MPJPE and PA-MPJPE for poses derived from the estimated meshes.SURREAL is a large-scale synthetic dataset with diverse body shapes and backgrounds; the experiments use its training set and test split.Source: §4.1 · p. 5

4.2. Implementation Details

  • The experiments use 64 markers learned on H3.6M and a shared marker set across datasets, with mixed training for H3.6M and 3DPW.A 3D pose estimator with HRNet-W48 estimates marker positions; the framework is trained for 40 epochs.Source: §4.2 · p. 5

4.3. Comparison to the State-of-the-arts

  • The method achieves state-of-the-art results on 3DPW and notably outperforms state-of-the-art methods on SURREAL, especially in MPVE; it is competitive or superior on H3.6M.The reported comparisons are proof-of-concept because methods may differ in backbones and training data.Source: §4.3 · p. 5, Table 2 · p. 6, Table 3 · p. 6

4.4. Ablation study

  • On H3.6M and SURREAL, learned virtual markers yield lower MPVE than skeleton or randomly sampled markers, supporting the learned representation over those alternatives.Randomly chosen vertices may be hard to detect or insufficiently expressive for reconstructing other vertices.Source: Table 4 · p. 6, §4.4 · pp. 6–7
  • With ground-truth marker positions, increasing marker count steadily lowers MPVE and MPJPE, but image-based MPVE begins rising when K exceeds 96.Some additional markers are difficult to detect from images and can introduce estimation errors.Source: §4.4 · p. 7, Table 5 · p. 7
  • Updating the coefficient matrix produces better mesh estimates than a fixed matrix, which incurs larger MPVE and MPJPE errors when markers are misestimated under occlusion.Back-side markers can be self-occluded, making their estimated positions inaccurate.Source: §4.4 · p. 7, Figure 6 · p. 7, Table 6 · p. 8

4.5. Qualitative Results

  • On natural and challenging images, the method generalizes across diverse postures, while local and sparse marker-to-vertex dependencies limit many occlusion or truncation errors to dependent regions.When body parts are truncated, the method still guesses positions for the truncated markers.Source: §4.5 · p. 8, Figure 8 · p. 8, Figure 7 · p. 8
  • The reported estimates for head orientation, facial expression, hands, and feet remain canonical because the datasets lack supervision for those parts.The authors attribute most other errors to inaccurate 3D marker estimates and suggest stronger estimators or more diverse training data as possible remedies.Source: §4.5 · p. 8

5. Conclusion

The conclusion presents Virtual Marker as a mesh representation that is more expressive than skeletons and more accessible than physical markers. The appendix discusses implementation and impact, while its post-processing section describes enforcing left-right symmetry.

5. Conclusion

  • Virtual Marker is more expressive than skeletons and more accessible than physical markers, supporting accurate and efficient mesh estimation, especially for diverse body shapes.Its coefficient matrix also encodes spatial relationships among mesh vertices, and the method shows generalization potential.Source: §5 · p. 8

Appendix

  • The appendix adds post-processing and experimental details and discusses human-subject data and potential societal impact.Source: Appendix · p. 11

A. Post-processing on Virtual Markers

B. Experiments

The experiments section introduces dataset descriptions and additional experimental results.

B. Experiments

  • The section first describes the datasets, then presents additional experimental results for the approach.Source: B. Experiments · p. 11

B.1 Datasets

The dataset descriptions specify the supervision and evaluation setup used across H3.6M, 3DPW, MPI-INF-3DHP, UP-3D, COCO, and SURREAL.

B.1 Datasets

  • H3.6M uses SMPL meshes fitted from MoSh markers for mesh supervision, with pose evaluated by MPJPE and PA-MPJPE and mesh error by MPVE.PA-MPJPE aligns estimates to ground-truth poses with Procrustes before computing MPJPE.Source: §B.1 · p. 11
  • 3DPW provides ground-truth SMPL parameters from IMU data when available; the model is trained on its train set and evaluated on its test set.Source: §B.1 · p. 11
  • MPI-INF-3DHP supplies 3D pose annotations but no mesh annotations, so supervision is enforced only on 3D skeletons in the mesh losses.Source: §B.1 · p. 11
  • UP-3D's fitted meshes are inaccurate, so its 2D annotations train the 3D virtual-marker estimation network instead.Source: §B.1 · p. 11
  • COCO's projected 3D mesh vertices align well with images, so its 2D annotations are used to train the virtual-marker estimation network.The passage notes that prior pseudo-SMPL mesh annotations from SMPLify-X are inaccurate.Source: §B.1 · p. 11
  • SURREAL provides synthetic humans with variation in shape, texture, viewpoint, and pose; the model is trained and tested on its respective splits.Source: §B.1 · p. 11

B.2 Implementation Details and Computation Resource

The implementation combines datasets and marker-based supervision for training, and the reported comparison emphasizes lower computation overhead alongside estimation quality.

B.2 Implementation Details and Computation Resource

  • For H3.6M and 3DPW, mix-training uses 2D and 3D datasets, and MPI-INF-3DHP's 17 landmark joints are added to the 64 virtual markers.Source: §B.2 · p. 11
  • The virtual-marker approach reduces computation overhead by a large margin while achieving better estimation quality than methods that directly regress all mesh vertices.Table 7 compares FLOPs and parameter counts, with MPVE on the 3DPW test set as performance reference.Source: Table 7 · p. 12, §B.2 · p. 12
  • The additional result tables cover H3.6M loss-term ablations and 3DPW virtual-marker occlusions, but the supplied captions do not report their outcomes.Source: Table 8 · p. 12, Table 9 · p. 12
  • The network is implemented in PyTorch and trained for 40 epochs with batch size 32 using Adam on four 16GB V100 GPUs.Source: §B.2 · p. 12

B.3 Additional Quantitative Results

Additional experiments examine loss terms, robustness to occlusion, and whether mesh fitting from detected virtual markers matches interpolation-based regression. The reported results support the marker estimates’ accuracy and the smoothing effect of surface losses.

Different loss terms

  • Ablating the surface losses increases MPVE substantially, while removing confidence loss increases it slightly on H3.6M.The authors attribute the surface losses’ effect to smoothing.Source: §B.3 · p. 12

Robustness to occlusion

  • Synthetic marker occlusion causes only a slight error increase overall, with larger increases when arm regions are occluded.The authors suggest the arm’s larger variations in the dataset may explain the greater increase.Source: §B.3 · p. 12

Comparison to fitting

  • 35.5 mm MPJPE over virtual markers; fitting a mesh to them gives similar error to interpolation-based regression on SURREAL.The comparison is intended to separate marker-detection accuracy from mesh-regression ability.Source: §B.3 · p. 12, Table 10 · p. 12

B.4 Additional Qualitative Results

Qualitative comparisons show advantages over skeleton-based and all-vertex representations on diverse shapes and challenging 3DPW cases. Internet-image results also show generalization alongside occlusion-related failures that tend to remain local.

B.4 Additional Qualitative Results

  • On diverse-shape SURREAL images, Pose2Mesh tends toward mean body shapes, while virtual markers yield more accurate shape estimates.The authors attribute Pose2Mesh’s behavior to the limited skeleton representation bottleneck.Source: §B.4 · p. 12
  • Internet-image examples include challenging cases with complex poses or extreme body shapes.The authors describe the method as generalizing well on natural scenes.Source: Figure 10 · p. 13, §B.4 · p. 13
  • Typical failures include inaccurate arm shape under occlusion, assigning another person’s lower arm to the subject, and right-hand interpenetration.The authors attribute these failures mainly to inaccurate virtual-marker estimates under occlusion; other body parts are barely affected.Source: Figure 11 · p. 13, §B.4 · p. 13
  • On 3DPW, the method produces smoother meshes and is more robust to truncation and occlusion than Pose2Mesh and METRO.The comparisons also report better image alignment; Pose2Mesh struggles with truncation or complex postures, while METRO outputs artifacts.Source: §B.4 · pp. 12–13

C. Human Subject Data

The experiments use public human-subject datasets under their official licensing requirements. The paper also notes a possible societal benefit, such as elderly fall detection.

C. Human Subject Data

  • Experiments use existing public human-subject datasets in accordance with their official licensing requirements.The authors state that proper use of the method could benefit society, for example through elderly fall detection.Source: C. Human Subject Data · p. 13
  • Qualitative comparisons include SURREAL, where the method is reported to improve estimates for diverse body shapes.The comparison is against Pose2Mesh.Source: Figure 12 · p. 14
  • On 3DPW, the qualitative comparison reports greater robustness to occlusion and truncation and more accurate image-aligned meshes.The comparison is against Pose2Mesh and METRO.Source: Figure 13 · p. 15
  • Additional qualitative meshes are shown for 3DPW, H3.6M, MPI-INF-3DHP, and COCO.Source: Figure 14 · p. 16

Related papers

Loading 2303.11726v4…