PaperlayerView exact revision on arXiv

Source-linked AI summary

FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration

Yu Rong, Takaaki Shiratori, Hanbyul Joo

arXiv:2108.06428v1cs.CV

TL;DR

Monocular whole-body 3D pose estimation is hindered by limited whole-body data and challenging face and hand observations. FrankMocap independently regresses face, hand, and body poses and integrates them into unified SMPL-X outputs using three latency–accuracy trade-offs. The system outperforms prior optimization-based and end-to-end methods, while its scope includes assumptions such as omitting SMPL-X eye poses and limited comparison access in some in-the-wild settings.

  • Problem

    Existing monocular 3D pose methods commonly focus on individual body parts, while estimating face, hands, and body together is difficult because whole-body 3D data are scarce and face and hand observations are challenging.

  • Method

    FrankMocap independently regresses SMPL-X face, hand, and body poses using separate modules, then combines them with one of three integration strategies.

  • Results

    FrankMocap outperforms existing optimization-based and end-to-end whole-body pose estimation methods on public benchmarks and in-the-wild scenarios.

  • Takeaways & Limitations

    The modular design supports interactive or offline use through integration methods that trade off latency and precision and can incorporate future part-estimation algorithms.

  • Takeaways & Limitations

    The system omits SMPL-X eye poses, and some in-the-wild comparisons are unavailable because one method’s code is private while another requires known camera intrinsics.

Abstract

from arXiv · show

Most existing monocular 3D pose estimation approaches only focus on a single body part, neglecting the fact that the essential nuance of human motion is conveyed through a concert of subtle movements of face, hands, and body. In this paper, we present FrankMocap, a fast and accurate whole-body 3D pose estimation system that can produce 3D face, hands, and body simultaneously from in-the-wild monocular images. The core idea of FrankMocap is its modular design: We first run 3D pose regression methods for face, hands, and body independently, followed by composing the regression outputs via an integration module. The separate regression modules allow us to take full advantage of their state-of-the-art performances without compromising the original accuracy and reliability in practice. We develop three different integration modules that trade off between latency and accuracy. All of them are capable of providing simple yet effective solutions to unify the separate outputs into seamless whole-body pose estimation results. We quantitatively and qualitatively demonstrate that our modularized system outperforms both the optimization-based and end-to-end methods of estimating whole-body pose.

1. Introduction

Whole-body 3D pose estimation is needed to capture subtle behavioral cues across the face, hands, and body, but small scales, occlusions, blur, and limited data make unified estimation difficult. FrankMocap addresses this challenge with independent part regressors followed by integration into a unified SMPL-X output.

  • Motivation: Whole-body motion estimation is necessary because human behavioral nuance is conveyed through subtle movements of the face, hands, and body.The paper connects whole-body pose estimation to applications including assistive technology, sign language understanding, AR/VR, and social signal understanding.
  • Challenges: Small faces and hands, viewpoint changes, self-occlusions, and motion blur make whole-body 3D pose estimation challenging.These factors also make large-scale whole-body 3D datasets difficult to create, including in controlled environments.
  • Existing gap: Limited whole-body 3D data has led most existing methods to estimate body, hand, or face pose separately rather than simultaneously.The paper identifies the lack of whole-body data as a major obstacle to constructing a unified estimator.
  • Proposed approach: FrankMocap independently regresses face, hand, and body poses, then integrates their outputs into a unified SMPL-X whole-body pose.The modular design is intended to exploit existing single-part regression performance while preserving accuracy and reliability in practice.
  • Contributions: Three integration strategies provide trade-offs between latency and precision, and the modular system outperforms optimization-based and end-to-end alternatives.The paper reports quantitative and qualitative evidence for this comparison.

2. Related Work

Prior work represents humans with parametric models and estimates body, hand, or face components from images, while FrankMocap combines separate component outputs into a whole-body system. Related approaches include optimization-based fitting and joint neural prediction of SMPL-X parameters.

  • 3D Parametric Human Body Models: Parametric human models represent face, hands, and body deformation using low-dimensional pose and shape parameters.Examples include SMPL for the body, MANO for hands, and whole-body models such as SMPL-X.
  • Single Image 3D Body, Hand and Face Estimation: Monocular 3D body methods estimate parametric-model parameters from RGB images using optimization, deep regression, or direct vertex regression.The body representation may use models such as SMPL or Adam.
  • Single Image 3D Body, Hand and Face Estimation: Hand and face estimation similarly uses neural networks to predict parametric-model parameters or directly regress hand meshes and face landmarks.Hand methods commonly use MANO, while face methods use models such as 3DMM or FLAME.
  • Terminology: In this paper, body denotes torso and limbs excluding finger joints, whereas whole-body denotes the face, hands, and body together.This terminology distinguishes body-only estimation from the complete whole-body setting.
  • Whole-body estimation systems: Figure 2 depicts FrankMocap applying separate part modules to a single RGB image before integrating their outputs into one whole-body result.This modular arrangement contrasts with ExPose, which jointly predicts body, hand, and face SMPL-X parameters using pseudo-ground-truth training data.

3. Method

FrankMocap estimates face, hands, and body independently, then integrates their outputs into a unified SMPL-X whole-body representation. Its modules use compatible parameterizations and alternative integration strategies to balance accuracy, robustness, and runtime.

  • Framework overview: Separate regressors estimate SMPL-X-compatible face, hand, and body parameters before integration produces a unified whole-body pose.The system processes both hands independently and combines all part outputs into a seamless whole-body result.
  • Whole-body parameterization: SMPL-X represents whole-body orientation, pose, facial expression, and shape with low-dimensional parameters and outputs a mesh plus regressed 3D joints.The model mesh contains 10,745 vertices, while whole-body joints are obtained by applying a joint-regression function.
  • Hand representation: The standalone hand model reuses SMPL-X hand parts, yielding 778 vertices and parameters directly compatible with whole-body integration.Its wrist, finger joints, and fingertips share the whole-body parameterization, enabling efficient combination with body outputs.
  • 3D hand estimation: The hand regressor maps a cropped RGB hand image to hand pose, shape, orientation, and weak-perspective camera parameters.An encoder-decoder network uses ResNet-50 features; left-hand inputs are vertically flipped and processed as right hands.
  • 3D hand estimation: Training combines pose, 3D-keypoint, 2D-keypoint, and regularization losses, with weights λ1 = 10, λ2 = 100, λ3 = 10 and λ4 = 0.1.The 2D-keypoint loss helps estimate camera parameters and supports generalization to images annotated only with 2D keypoints.
  • Whole-body integration: Integration transfers compatible joint angles while resolving conflicting body- and hand-derived wrist poses; optimization additionally uses reprojection, mesh, and prior terms.The body module is adapted from SPIN and fine-tuned with SMPL-X, while RingNet supplies facial expressions and face poses.

4. Experiments

Experiments evaluate FrankMocap’s hand and body modules, integration variants, and whole-body performance against prior methods on public benchmarks and challenging in-the-wild scenes.

  • Hand Module Evaluation: The hand module is comparable with previous state-of-the-art hand-only methods on STB, RHD, and MPII+NZSL benchmarks.Evaluation uses PCK AUC, with 3D thresholds of 20–50 mm for STB and RHD and 2D thresholds of 0–30 px for MPII+NZSL.
  • Hand Module Evaluation: Data augmentation improves hand-module results and generalization to blur, pose variations, and occlusion.The ablation compares models trained with different augmentation strategies, including motion-blur augmentation.
  • Body Module Evaluation: The body module is comparable to state-of-the-art body-only methods, including ExPose, on the 3DPW test set.The module is a fine-tuned version of SPIN and EFT; its slightly degraded performance may reflect using fewer datasets.
  • Body Module Evaluation: The body-module comparison is limited because Kulton et al.’s code is unavailable for additional challenging in-the-wild comparisons.CMR also requires known camera intrinsics, preventing application to in-the-wild scenarios.
  • Integration Module Evaluation: All FrankMocap integration variants significantly outperform SMPLify-X and ExPose on EHF whole-body, body, and hand estimation.The optimization-based variant generally performs best, while wrist integration outperforms copy-paste; face estimation is comparable with prior methods.
  • Integration Module Evaluation: FrankMocap with wrist integration outperforms MTC, SMPLify-X, and ExPose qualitatively in challenging in-the-wild scenes.The comparison is reported for fast and accurate whole-body motion capture, with additional qualitative results in the appendix.

5. Conclusion

FrankMocap combines independent state-of-the-art part estimators with integration modules to produce monocular whole-body pose estimates. Its variants span speed and precision, while the modular design supports future module replacement.

  • Conclusion: FrankMocap uses regression-and-integration to combine independent face, hand, and body estimators into whole-body pose outputs.The design leverages existing single-part pose datasets and avoids dependence on large whole-body pose datasets.
  • Conclusion: The copy-paste, wrist-integration, and optimization-based modules trade off latency and precision for different application settings.Copy-paste and wrist integration suit time-sensitive applications, whereas optimization suits offline applications requiring better precision.
  • Conclusion: FrankMocap surpasses previous state-of-the-art 3D whole-body estimation methods on public benchmarks and in-the-wild scenarios.Its modular design can leverage future algorithms by replacing individual modules.

A1. Hand Module

This appendix section introduces the stand-alone hand module, covering its mesh, data preprocessing, implementation, datasets, and experiments on dataset choices.

  • A1. Hand Module: The hand-module appendix covers hand-mesh hierarchy, cross-dataset preprocessing, implementation details, hand datasets, and dataset-effect experiments.The stated goal is to document the stand-alone hand module and evaluate how different datasets affect it.

A1.1. Hand Module Details

The hand module visualizes the cropped hand mesh and hand-skeleton hierarchy, and preprocesses multiple public datasets to make their annotations compatible with the hand model.

  • Figure 10 depicts the cropped hand mesh and the hierarchy of the hand skeleton.
  • The module includes as many publicly available datasets as possible to support in-the-wild generalization.
  • Preprocessing resolves annotation discrepancies across datasets by making them consistent and compatible with the hand model.

A1.2. Hand Datasets

The hand module is trained and evaluated using a diverse collection of real, synthetic, interaction-focused, and in-the-wild datasets. The paper also studies how dataset composition affects hand-pose generalization.

  • Datasets: FreiHAND, HO-3D, MTC, STB, and RHD are used to train the hand module, while MPII+NZSL is reserved for validation.The training set combines five datasets; MPII+NZSL contains challenging occlusion, blur, and low-resolution images and is not used for training.
  • Datasets: HO-3D focuses on hand-object interaction and supplies 3D joints and MANO pose parameters for hands.
  • Evaluation: The dataset ablation evaluates hand models on MPII+NZSL using 2D AUC, comparing different training-data combinations.The tested combinations include FreiHAND plus HO-3D, STB plus RHD plus MTC, and the full dataset collection.
  • Datasets: STB and RHD provide annotated hand data, with STB used for training and validation comparisons and RHD supplying synthetic 2D and 3D joint annotations.STB contains 15,000 training and 3,000 testing samples; RHD contains 41,258 training and 2,728 testing samples.

A1.3. Ablation Study on Hand Datasets.

The hand-dataset ablation finds that incorporating more datasets improves quantitative performance and produces more precise results on in-the-wild images.

  • Quantitative ablation: Using more datasets leads to better hand-module performance in the quantitative ablation.
  • Qualitative ablation: The full dataset combination improves qualitative generalization and precision on in-the-wild images relative to smaller dataset subsets.The compared subsets are FreiHAND plus HO-3D, STB plus RHD plus MTC, and the full set.

A2.1. Framework Details.

The framework combines independently estimated body and hand information through wrist integration, with qualitative results and failure cases illustrating its practical behavior. The system is affected by severe occlusion and blur, while comparisons report more accurate hand and body poses than Zhou et al.

  • Wrist integration: The wrist integration network adjusts arm poses so the hands move to locations predicted by the hand module.It takes a 2D image-space arm-direction vector and arm pose parameters from the copy-paste integration output.
  • Wrist integration: Shoulder orientations are converted to global coordinates while elbow poses remain local rotations to simplify wrist-integration training.This transformation removes the need to account for other body-part poses when processing the shoulder orientation.
  • Network design: The wrist integration network is a six-layer MLP mapping an eight-dimensional input to six arm-pose outputs.Its input contains arm poses and a 2D directional vector; intermediate layer widths are 128, 256, 512, 256, and 128.
  • Inference: At inference, left-arm inputs and directional vectors are vertically flipped into right-arm space and flipped back after prediction.
  • Failure cases: Typical FrankMocap failure cases are caused by severe occlusion and blurry inputs.
  • Qualitative evaluation: Qualitative comparison with Zhou et al. reports more accurate hand and body poses, although their official code was unavailable and paper images were used.
Loading 2108.06428v1…