PaperlayerView exact revision on arXiv

Source-linked AI summary

Monocular Total Capture: Posing Face, Body, and Hands in the Wild

Donglai Xiang, Hanbyul Joo, Yaser Sheikh

arXiv:1812.01598v1cs.CVcs.GR

TL;DR

Monocular total-motion capture remains difficult because depth ambiguity and limited in-the-wild 3D data constrain existing methods. This paper combines POF-based CNN prediction, deformable-model fitting, temporal texture tracking, and a 40-subject dataset, demonstrating total capture in challenging videos and competitive benchmark performance. Its practical scope is limited by occlusion, low resolution, motion blur, required bounding boxes, and inability to process multiple people or hands simultaneously.

  • Problem

    Monocular total-motion capture lacks paired in-the-wild 3D data and must overcome depth ambiguity while reconstructing body, hands, and face together.

  • Method

    The method predicts 3D Part Orientation Fields and joint confidence maps, fits a deformable human model, and enforces photometric consistency across frames.

  • Results

    The framework achieves comparable results to state-of-the-art 3D body and hand methods and demonstrates total-motion capture on challenging in-the-wild videos.

  • Takeaways & Limitations

    A single framework can reconstruct total motion from monocular images or videos and provide training and evaluation data through a 40-subject dataset.

  • Takeaways & Limitations

    The method fails with substantial occlusion, insufficient hand-image resolution, or severe motion blur, and requires bounding boxes while handling only one person and one set of hands.

Abstract

from arXiv · show

We present the first method to capture the 3D total motion of a target person from a monocular view input. Given an image or a monocular video, our method reconstructs the motion from body, face, and fingers represented by a 3D deformable mesh model. We use an efficient representation called 3D Part Orientation Fields (POFs), to encode the 3D orientations of all body parts in the common 2D image space. POFs are predicted by a Fully Convolutional Network (FCN), along with the joint confidence maps. To train our network, we collect a new 3D human motion dataset capturing diverse total body motion of 40 subjects in a multiview system. We leverage a 3D deformable human model to reconstruct total body pose from the CNN outputs by exploiting the pose and shape prior in the model. We also present a texture-based tracking method to obtain temporally coherent motion capture output. We perform thorough quantitative evaluations including comparison with the existing body-specific and hand-specific methods, and performance analysis on camera viewpoint and human pose changes. Finally, we demonstrate the results of our total body motion capture on various challenging in-the-wild videos. Our code and newly collected human motion dataset will be publicly shared.

1. Introduction

The paper addresses the difficulty and limited accessibility of monocular 3D human motion capture, especially for complete body, hand, and face motion in the wild. It introduces a unified framework, dataset, and temporal refinement strategy for this task.

  • Motivation: Monocular 3D motion capture could make human-motion data more accessible than multi-camera or depth-camera systems.Existing systems are costly and require specialized capture setups, limiting available 3D motion data.
  • Challenges: Depth ambiguity and scarce paired in-the-wild image–3D datasets make monocular body and hand pose estimation difficult.Previous methods consequently demonstrate most results in controlled laboratory environments.
  • Research gap: No prior method reconstructed body, hands, and face motion together from a single view, despite the importance of complete human behavior understanding.
  • Approach: The paper introduces Part Orientation Fields to encode 3D body-part orientations in 2D image space for monocular total-motion capture.
  • Contributions: The framework fits a deformable human model to 3D POFs and 2D keypoints, achieving comparable results to state-of-the-art body and hand methods.
  • Contributions: The paper contributes photometric temporal refinement and a new 3D motion dataset containing 40 subjects.

2. Related Work

Related work spans monocular 2D and 3D body estimation, hand pose estimation, and deformable human models. These lines of research provide key components but are generally specialized by body part, sensing modality, or model scope.

  • Single-image 2D pose estimation: Fully convolutional networks commonly estimate 2D body joints using confidence heatmaps, which are more efficient than direct coordinate regression.
  • Single-image 3D pose estimation: Monocular 3D body methods either lift detected 2D poses or directly estimate 3D pose, with two-stage approaches discarding image information.
  • Single-image 3D pose estimation: Human-pose priors have been incorporated through ordinal depth, bone-length and joint-angle constraints, and GAN-based modeling.
  • Monocular hand pose estimation: Hand pose estimation has largely used depth input, while RGB methods for 2D and 3D hand pose emerged more recently.
  • 3D deformable human models: Deformable human models constrain reconstruction to parametric shape and motion spaces, simplifying markerless capture while limiting expressiveness.

3. Method Overview

The method processes monocular image sequences through CNN prediction, deformable-mesh fitting, and temporal refinement. It represents total motion with shared sequence shape parameters and frame-specific pose, scale, and translation parameters.

  • Input and output: The system outputs body, face, hand, and foot motion for a single person as a deformable 3D human model in every frame.
  • Model parameters: Shape coefficients φ are shared across the sequence, while pose θ, scale σ, and translation t are estimated separately for each frame.
  • CNN prediction: The first stage predicts joint confidence maps and 3D Part Orientation Fields from each image using a convolutional neural network.
  • Mesh fitting: The second stage fits a deformable human mesh to CNN image measurements, using the model prior to improve robustness against prediction noise.
  • Temporal refinement: The third stage enforces photometric consistency in mesh textures across frames to reduce motion jitters.
  • Temporal refinement: The authors report that temporal refinement is crucial for realistic body-motion capture output.

4. Predicting 3D Part Orientation Fields

The method represents each articulated body part with a 3D Part Orientation Field (POF), encoding its orientation across the corresponding region of the 2D image. A CNN predicts these orientation fields together with joint confidence maps.

  • A POF represents the 3D orientation of a body part connecting parent joint Jm to child joint Jn in camera coordinates.The orientation is the unit vector from Jm to Jn.
  • Figure 3 illustrates that all pixels assigned to a body part receive the same unit orientation vector in the POF's x, y, and z channels.The vector points from the parent joint to the child joint.
  • Each POF is a three-channel heatmap whose channels encode the orientation's x, y, and z components over the image.The field has spatial dimensions matching the image height and width.
  • POF values are assigned only to pixels belonging to the corresponding body-part region, represented using a rectangular part mask.Pixels outside the current target part do not receive that part's orientation value.
  • The CNN is trained to predict joint confidence maps S and Part Orientation Fields L from a cropped 368×368 target-person image.During testing, the crop is obtained from an OpenPose bounding box.

5. Model-Based 3D Pose Estimation

The method fits a deformable human mesh model to CNN-predicted 2D joints and POFs, using pose and shape priors to estimate body, hand, foot, and face motion. It then optimizes the combined objectives in stages to recover total-body motion parameters.

  • Deformable Mesh Model Fitting with POFs: CNN outputs are used to reconstruct a 3D skeletal structure up to scale, while deformable-model priors improve robustness to noisy measurements.The model includes expressive body, hand, and facial parameters through Adam.
  • Deformable Mesh Model Fitting with POFs: The body fitting process extracts 2D joint locations from confidence maps and averages POF values along each image-space bone segment.These measurements constrain the mesh parameters θ, φ, and t.
  • Deformable Mesh Model Fitting with POFs: The optimization combines 2D keypoint, POF, and pose-prior terms to align projected model joints with detected image evidence and feasible human poses.The POF term penalizes disagreement between predicted orientations and mesh-bone directions; pose priors are learned from CMU Mocap.
  • Deformable Mesh Model Fitting with POFs: Levenberg-Marquardt optimization fits the mesh, and Figure 4 summarizes the sequence from confidence-map joints and POF orientations to model fitting.The figure highlights minimizing the distance between projected mesh quantities and estimated POF orientations.
  • Total Body Capture with Hands, Feet and Face: Hand fitting uses analogous objectives for both hands, with hand pose priors learned from the MANO dataset.The left- and right-hand objectives are optimized as parts of the total-body model.
  • Total Body Capture with Hands, Feet and Face: Face and foot keypoints from OpenPose add constraints that extend the fitted body-and-hand model to toes and facial parameters.Facial keypoints depend jointly on body pose, shape, translation, and expression parameters.
  • Total Body Capture with Hands, Feet and Face: The final objective combines these constraints and is optimized in multiple stages, fitting the torso first, then limbs, and finally all terms.This produces per-frame Adam model parameters for 3D total-body motion.

6. Enforcing Photo-Consistency in Textures

The method enforces temporal consistency by extracting mesh textures and using optical flow to align projected vertices across frames. It combines this photometric constraint with POF and face-keypoint terms while recursively updating frame parameters and fixing sequence-level shape.

  • Texture extraction and optical flow: Texture maps are extracted from visible mesh regions and reused to compare consecutive frames through a rendered synthetic image.The previous frame’s texture is applied to the next frame’s mesh rendering before comparison with the next input image.
  • Texture extraction and optical flow: Optical flow maps projected mesh vertices toward locations that improve photometric texture consistency across frames.The flow destination for each vertex is pre-computed and remains constant during optimization.
  • Jitter reduction: The image-space consistency term mainly reduces x- and y-direction jitter, while a separate smoothness constraint regularizes z-coordinate changes.The method adds depth smoothness because image evidence does not directly constrain z-direction jitter.
  • Recursive optimization: The tracking objective replaces most 2D joint constraints with the texture term but retains POF and face-keypoint terms to avoid error accumulation.Optimization starts from a good per-frame initialization and uses the updated parameters recursively for subsequent frames.
  • Recursive optimization: Shape parameters remain fixed across the sequence, and facial expression is frozen during the texture-based optimization stage.The fixed shape is taken from the preceding optimized frame while facial expression is not updated.

7. Results

The evaluation compares the method on body and hand benchmarks, studies viewpoint, pose, and temporal refinement, and demonstrates qualitative capture across indoor and in-the-wild imagery and videos.

  • Benchmark evaluations: The method is evaluated on Human3.6M, STB, and Dexter+Object using established body- and hand-pose protocols.Human3.6M uses root-aligned MPJPE; STB and Dexter+Object use hand-pose evaluation procedures including PCK/AUC settings.
  • 3D body pose estimation: Our method achieves the lowest pose estimation error among methods reporting results on in-the-wild videos in the Human3.6M comparison.The authors also report competitive overall performance, while pose prior increases error despite improving surface structure and joint angles in the wild.
  • Ablation studies: Adding the newly captured dataset improves Human3.6M reconstruction, and adding COCO produces a further error decrease.The ablation compares Human3.6M alone, Human3.6M plus the captured dataset, and both datasets plus COCO.
  • 3D hand pose estimation: AUC=0.84 with ground-truth root-depth alignment outperforms the previous state-of-the-art at AUC=0.70 and an RGB-D method at AUC=0.81.With fixed hand scale instead, the method reaches AUC=0.70 versus AUC=0.71 for the compared state-of-the-art.
  • Viewpoint and pose changes: 6.30 cm average MPJPE is reported on the newly collected multi-view dataset, with higher errors under elevated, back-facing, self-occluded, or foreshortened configurations.The study groups errors by camera viewpoint and body-pose clusters.
  • Temporal and qualitative evaluation: Temporal refinement makes reconstructed motion more stable and often closer to ground truth, while qualitative results cover indoor Mocap images and self-recorded or YouTube videos.For upper-body-only videos, the method assumes torso and leg orientations point vertically downward.

8. Discussion

The method reconstructs a single person’s 3D total motion from an image or monocular video and is evaluated on challenging in-the-wild videos. Its main limitations concern visibility, image quality, and input scope.

  • The method simultaneously reconstructs 3D total motion from a single image or monocular video.
  • Qualitative results present predictions for in-the-wild images with zoomed, side, and top views.
  • Significant occlusion or out-of-frame body regions can cause failures through erroneous network predictions.
  • The hand pose detector fails under insufficient resolution or severe motion blur.
  • The CNN requires body and hand bounding boxes and cannot process multiple bodies or hands simultaneously.

A. New 3D Human Pose Dataset

The paper introduces a new 3D human pose dataset and provides a section describing its construction.

  • The dataset is newly collected for 3D human pose research.
  • The section focuses on details of the dataset rather than presenting its construction in this passage.
  • Dataset construction is treated as a distinct section of the paper.

A.1. Methodology

The dataset is built from multiview recordings of 40 volunteers, reconstructed and filtered into body, hand, and face annotations. It supplies hundreds of thousands of image-annotation pairs for training networks.

  • Methodology: The dataset recruits 40 volunteers and captures their motion in a multiview system while they follow a prerecorded video.
  • Methodology: Multi-view 3D reconstruction algorithms produce 3D body, hand, and face keypoints.
  • Methodology: Filtering removes frames with inconsistent bone lengths, followed by manual verification of hand annotations across three camera views.
  • Methodology: Training uses 834K body image-annotation pairs and 111K hand image-annotation pairs.
  • Methodology: The representation predicts body and hand joints together with Part Orientation Fields at 368 × 368 spatial resolution.

C.1. Model Parameters

The model represents total-body motion through shape, pose, translation, and facial-expression parameters, with network and OpenPose measurements mapped to model keypoints for fitting.

  • Model Parameters: The model parameters comprise shape, pose, global translation, and facial-expression variables.
  • Model Parameters: The shape deformation space has dimension Kφ = 30, while the model pose uses J = 62 joints and facial expression uses Kσ = 200 bases.
  • Model Parameters: Body regressors compute model joints from mesh vertices for fitting body pose.
  • Model Parameters: Hand regressors define hand keypoints and parts used to fit hand pose.
  • Model Parameters: The model contains 22 body joints and 20 joints for each hand.
  • Model Parameters: OpenPose foot keypoints are matched to mesh vertices for toes and constructed heel points.
  • Model Parameters: Facial expression is fitted using 41 OpenPose keypoints on the eyes, nose, mouth, and eyebrows, excluding face-contour points.

D. Implmentation Details

The implementation details specify CNN outputs and mesh-fitting keypoints, together with weighting choices for the optimization terms. Figures illustrate confidence maps and the x, y, z channels of predicted Part Orientation Fields for body and hand images.

  • Mesh fitting: Adam mesh vertices serve as keypoints for fitting both feet and facial expression.The figure marks the selected vertices with red dots; the middle points between the two rear foot vertices are also used as keypoints.
  • POF objective: The combined FPOF objective contains POF terms for the body, left hand, and right hand, weighted 25, 1, and 1, respectively.These weights balance the three terms in Equation 16.
  • CNN outputs: The CNN predicts joint confidence maps for body and hand images.Separate figures visualize the predicted confidence maps for each image type.
  • CNN outputs: The CNN predicts Part Orientation Fields for body and hand images, visualizing each part’s x, y, and z channels separately.The visualizations show the three orientation channels for each body part.
Loading 1812.01598v1…