PaperlayerView exact revision on arXiv

Source-linked AI summary

Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation

Gyeongsik Moon, Hongsuk Choi, Kyoung Mu Lee

arXiv:2011.11534v4cs.CV

TL;DR

Whole-body 3D mesh estimation seeks to reconstruct the body, hands, and face together, but accurate hand recovery remains challenging because wrist prediction does not adequately use kinematic cues and body features provide coarse finger information. Hand4Whole uses Pose2Pose and MCP-joint features for wrist rotations while omitting body features for finger rotations. The authors report that it outperforms previous whole-body methods on all benchmarks.

  • Problem

    Accurate whole-body hand recovery remains challenging because wrist prediction underuses kinematic-chain cues and body features contain coarse hand information for predicting fingers.

    Source: §1 · pp. 1–2

  • Method

    Hand4Whole uses Pose2Pose to combine body and MCP-joint features for wrist rotations and omits body features when predicting finger rotations.

    Source: §1 · p. 2

  • Results

    Hand4Whole largely outperforms previous 3D whole-body human mesh estimation methods on all benchmarks.

    Source: §1 · pp. 2–3, §6 · p. 8

  • Takeaways & Limitations

    The paper concludes that using MCP-joint features for wrist rotations and excluding body features for finger rotations produces better 3D hands in whole-body meshes.

    Source: §6 · p. 8

Abstract

from arXiv · show

Whole-body 3D human mesh estimation aims to reconstruct the 3D human body, hands, and face simultaneously. Although several methods have been proposed, accurate prediction of 3D hands, which consist of 3D wrist and fingers, still remains challenging due to two reasons. First, the human kinematic chain has not been carefully considered when predicting the 3D wrists. Second, previous works utilize body features for the 3D fingers, where the body feature barely contains finger information. To resolve the limitations, we present Hand4Whole, which has two strong points over previous works. First, we design Pose2Pose, a module that utilizes joint features for 3D joint rotations. Using Pose2Pose, Hand4Whole utilizes hand MCP joint features to predict 3D wrists as MCP joints largely contribute to 3D wrist rotations in the human kinematic chain. Second, Hand4Whole discards the body feature when predicting 3D finger rotations. Our Hand4Whole is trained in an end-to-end manner and produces much better 3D hand results than previous whole-body 3D human mesh estimation methods. The codes are available here at https://github.com/mks0601/Hand4Whole_RELEASE.

1. Introduction

Whole-body mesh estimation must recover hands despite their small image size and complex articulation. Hand4Whole combines body and MCP-joint cues for wrist rotations while omitting body features for finger rotations, and the authors report that it outperforms previous whole-body methods.

1. Introduction

  • Whole-body mesh estimation reconstructs body, hands, and face together, with small hand image regions and complex articulation making accurate hand recovery challenging.The paper also identifies smoothly connecting the estimated body and hand meshes as a challenge.Source: §1 · pp. 1–2
  • Pose2Pose predicts wrist rotations using body features and eight hand MCP-joint features, combining overall posture with joint-specific wrist cues.The body cue supports plausible wrist rotations when hands are occluded or truncated; MCP joints provide essential rotation information.Source: §1 · p. 2, Figure 2 · p. 2
  • Hand4Whole significantly outperforms previous whole-body 3D human mesh estimation methods while being trained end-to-end.The paper reports better hand results but supplies no benchmark values in these passages.Source: §1 · pp. 2–3
  • Hand4Whole omits body features for finger rotations to avoid coarse hand information, while body-aware wrist rotations preserve anatomically plausible connections.The wrists are root nodes in the hand kinematic chain and determine global finger rotations.Source: §1 · p. 2

2. Related works

Prior work spans optimization-based fitting and regression-based networks for whole-body mesh estimation. Hand4Whole distinguishes itself by using joint-specific features and combining body and MCP cues for wrists, while excluding body features from finger prediction.

2. Related works

  • Optimization-based methods fit human models to 2D or 3D evidence, but can be slow and prone to noisy evidence.The related-work section describes regression-based approaches as a recent alternative.Source: §2 · p. 3
  • Regression methods generally use separate body, hand, and face networks, with ExPose, FrankMocap, Zhou et al., and PIXIE differing in their network arrangements and rotation cues.ExPose uses three networks, FrankMocap uses body and hand networks, Zhou et al. use 3D joint coordinates, and PIXIE adds a moderator for rotations.Source: §2 · p. 3
  • Unlike prior systems that use global pooled instance features, Hand4Whole uses joint features and body-plus-MCP cues for wrists, then excludes body features for finger rotations.The cited comparisons note that hand-only wrist predictions can be implausible under occlusion, body-only predictions lack MCP cues, and coarse hand information can hurt wrist prediction.Source: §2 · p. 3

3. Hand4Whole

Hand4Whole combines BodyNet, HandNet, and FaceNet to produce a whole-body mesh, with Pose2Pose guiding joint rotations from joint-specific features. Its design uses hand MCP information for body rotations and detailed hand crops for finger rotations.

3.1. Pose2Pose

  • Pose2Pose estimates 3D joint coordinates, samples joint-specific features from a convolved image map, and uses both to predict 3D joint rotations.It predicts coordinates from 3D heatmaps and extracts features by bilinear interpolation at each joint’s pixel position.Source: §3.1 · pp. 3–4

3.2. BodyNet

  • BodyNet predicts body rotations, shape and camera parameters, and hand and face boxes, using hand MCP coordinates and features to improve body rotations.The MCP information also supports elbow rotation prediction; the body image feature supplies shape and translation parameters.Source: §3.2 · p. 4

3.3. HandNet

  • HandNet predicts finger rotations from high-resolution hand crops and supplies features and coordinates from eight MCP joints to BodyNet.The left-hand crops are flipped before processing, and their outputs are flipped back afterward.Source: §3.3 · p. 5

3.4. FaceNet

  • FaceNet predicts jaw rotation and facial expression from a face crop using a GAP-based regressor rather than Pose2Pose.The face crop comes from the high-resolution image at the box predicted by BodyNet.Source: §3.4 · p. 5

3.5. Loss functions

  • End-to-end training minimizes a loss combining parameter, joint-coordinate, and hand/face box errors.The coordinate loss covers predicted joint coordinates, SMPL-X-regressed mesh joints, and projected 2D joints; the box loss covers centers and scales.Source: §3.5 · p. 5

4. Implementation details

The implementation uses an Adam optimizer, body-pose pretraining, and standard image augmentations, with a neutral-gender SMPL-X model for training and testing.

4. Implementation details

  • Training uses Adam with batch size 96, a 10^-4 initial learning rate, and a tenfold reduction at epoch 10.Augmentations include scaling, rotation, horizontal flips, and color jittering; rotations are predicted in a 6D representation and converted to axis-angle.Source: §4 · p. 5
  • The body ResNet is initialized from Xiao et al. [33], pretrained on MSCOCO 2D human body pose, and a neutral-gender SMPL-X model is used for training and testing.The implementation uses PyTorch.Source: §4 · p. 5

5. Experiment

Hand4Whole performs strongly across whole-body, body-only, and hand-only benchmarks, with particularly accurate hand predictions. Ablations examine wrist inputs, finger features, and joint-feature combinations.

5.1. Datasets and evaluation metrics

  • The evaluation uses Human3.6M, whole-body MSCOCO, MPII, and FreiHAND for training, with separate benchmarks for body, hand, face, and whole-body evaluation.MPJPE and MPVPE measure joint and vertex distances after root-translation alignment; PA variants additionally align rotation and scale.Source: §5.1 · p. 5

5.2. Ablation study

  • Body-plus-MCP features yield the lowest 3D hand errors and accurate wrist rotations among the tested wrist-feature variants.The compared variants use body features alone, body plus a hand GAP feature, all hand-joint features, or body plus MCP-joint features.Source: §5.2 · p. 6, Table 1 · p. 6
  • Removing coarse hand information from the body feature improves 3D finger rotations, while the coarse feature contains unnecessary body and background information but little finger information.The comparison evaluates hand PA MPVPE with and without the coarse hand information.Source: Table 2 · p. 6, §5.2 · p. 7
  • Combining 3D joint coordinates with joint features achieves the best whole-body PA MPVPE among the tested input combinations.The reported comparison also finds lower PA MPVPE with 3D rather than 2D joint coordinates.Source: Table 3 · p. 6, §5.2 · p. 7

5.3. Comparison with state-of-the-art methods

  • Figure 6 shows more accurate 3D wrist and finger rotations than previous whole-body methods.Hand4Whole is also reported to outperform previous whole-body methods on 3DPW, though recent body-only methods slightly beat it there, and on FreiHAND, though recent hand-only methods slightly beat it there.Source: Table 6 · p. 7, Table 7 · p. 7, §5.3 · p. 8
  • Hand4Whole largely outperforms previous whole-body methods across EHF, AGORA, 3DPW, and FreiHAND.The authors report slight advantages for recent body-only methods on 3DPW and hand-only methods on FreiHAND.Source: §5.3 · pp. 7–8, Table 4 · p. 7, Table 5 · p. 7

6. Conclusion

The conclusion presents Hand4Whole as a whole-body mesh estimator with improved 3D hand results. It attributes wrist prediction to Pose2Pose with hand MCP features and finger prediction to omitting body features.

6. Conclusion

  • Hand4Whole uses Pose2Pose with hand MCP features for wrist rotations, omits body features for finger rotations, and outperforms prior whole-body methods on all benchmarks.The conclusion describes its 3D hand results as much better within whole-body meshes.Source: §6 · p. 8

Supplementary Material for “Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation”

The supplementary material presents additional experimental results that the main manuscript omitted for lack of space.

Supplementary Material

A.1. MSCOCO

Supplementary MSCOCO figures show more accurate in-the-wild results for Hand4Whole, particularly for 3D hands.

A.1. MSCOCO

  • Figures A and B show more accurate Hand4Whole results on in-the-wild MSCOCO images, particularly for 3D hands.The excerpt does not report quantitative comparisons.Source: §A.1 · p. 10

A.2. 3DPW

On 3DPW videos, Hand4Whole produces more accurate and plausible expressive 3D human meshes than ExPose and FrankMocap. Its hand results are more stable when hands are invisible; all methods run frame by frame without temporal information or postprocessing.

A.2. 3DPW

  • On 3DPW videos, Hand4Whole produces more accurate and plausible expressive 3D human meshes than ExPose [4] and FrankMocap [27].The comparison methods use their officially released codes.Source: §A.2 · p. 10
  • When hands are invisible, Hand4Whole achieves much better and more stable hand results using body and hand MCP joint features.Source: §A.2 · p. 10
  • Hand4Whole, ExPose, and FrankMocap run on each frame without temporal information or postprocessing such as average filtering.Source: §A.2 · p. 10

License of the Used Assets

The listed datasets and codebases carry a mix of academic-use restrictions and Creative Commons licenses. Two qualitative figures compare Hand4Whole with ExPose, FrankMocap, and PIXIE on MSCOCO.

License of the Used Assets

  • Human3.6M, MPII, 3DPW, FreiHAND, Stirling, EHF, and AGORA are listed as available for academic research only, with non-commercial conditions stated for several datasets.Human3.6M is limited to academic use; MPII, 3DPW, FreiHAND, Stirling, EHF, and AGORA are described as free to educational or research institutes for non-commercial research.Source: License of the Used Assets · p. 10
  • MSCOCO is licensed under Creative Commons Attribution 4.0, while the FFHQ dataset is made available under Creative Commons BY-NC-SA 4.0.FFHQ images were published under several possible licenses, including Creative Commons BY 2.0, BY-NC 2.0, Public Domain Mark 1.0, CC0 1.0, or U.S. Government Works.Source: License of the Used Assets · p. 10
  • ExPose and PIXIE codes are listed as academic-research-only, while FrankMocap codes are CC-BY-NC 4.0 licensed.Source: License of the Used Assets · p. 10
  • Figures A and B show qualitative MSCOCO comparisons of Hand4Whole with ExPose, FrankMocap, and PIXIE.Source: License of the Used Assets · p. 11
Loading 2011.11534v4…