PaperlayerView exact revision on arXiv

Source-linked AI summary

WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion

Soyong Shin, Juyong Kim, Eni Halilaj, Michael J. Black

arXiv:2312.07531v2cs.CV

TL;DR

Existing video methods struggle with global coordinates, non-flat terrain, computational cost, and per-frame accuracy. WHAM lifts 2D keypoint sequences using motion-capture data, fuses them with video features, and uses camera angular velocity and foot contact for global trajectory refinement. It achieves state-of-the-art performance across multiple in-the-wild benchmarks while maintaining temporal smoothness and efficient online inference.

  • Problem

    3D human motion recovery from video remains limited by camera-coordinate estimates, flat-ground assumptions, foot sliding, expensive optimization, and lower video accuracy than single-frame methods.

  • Method

    WHAM lifts 2D keypoint sequences to 3D with motion-capture data, fuses motion and video features, and refines global trajectories using camera angular velocity and foot contact.

  • Results

    WHAM achieves state-of-the-art performance on multiple in-the-wild benchmarks, including per-frame accuracy that surpasses image- and video-based methods while maintaining temporal smoothness.

  • Takeaways & Limitations

    WHAM provides fast, accurate global 3D human motion recovery from moving-camera video, including motion on non-planar surfaces.

  • Takeaways & Limitations

    WHAM’s motion generalization is limited for out-of-distribution motions, and its data synthesis mainly assumes the full body remains within the camera’s field of view.

Abstract

from arXiv · show

The estimation of 3D human motion from video has progressed rapidly but current methods still have several key limitations. First, most methods estimate the human in camera coordinates. Second, prior work on estimating humans in global coordinates often assumes a flat ground plane and produces foot sliding. Third, the most accurate methods rely on computationally expensive optimization pipelines, limiting their use to offline applications. Finally, existing video-based methods are surprisingly less accurate than single-frame methods. We address these limitations with WHAM (World-grounded Humans with Accurate Motion), which accurately and efficiently reconstructs 3D human motion in a global coordinate system from video. WHAM learns to lift 2D keypoint sequences to 3D using motion capture data and fuses this with video features, integrating motion context and visual information. WHAM exploits camera angular velocity estimated from a SLAM method together with human motion to estimate the body's global trajectory. We combine this with a contact-aware trajectory refinement method that lets WHAM capture human motion in diverse conditions, such as climbing stairs. WHAM outperforms all existing 3D human motion recovery methods across multiple in-the-wild benchmarks. Code will be available for research purposes at http://wham.is.tue.mpg.de/

1. Introduction

WHAM targets accurate, efficient 3D human pose and shape recovery in global coordinates from monocular video, including moving cameras and non-planar surfaces. It combines learned motion context with visual information, camera motion, and foot contact to achieve accurate and temporally smooth reconstruction.

  • Motivation: WHAM addresses global-coordinate recovery, computational efficiency, temporal smoothness, realistic foot-ground contact, and arbitrary moving cameras for monocular 3D human capture.These goals support applications including gaming, AR/VR, autonomous driving, sports analysis, and human-robot interaction.
  • Motivation: Existing video-based 3D human pose and shape methods are less accurate than the best single-frame methods, partly reflecting limited video datasets with ground-truth 3D poses.Single-image datasets provide broader variation in body shapes, poses, backgrounds, and lighting than available video datasets.
  • Approach: WHAM pretrains motion encoders and decoders on AMASS-generated 2D keypoint and 3D motion pairs, then fuses motion context with dense video features for pose and shape estimation.Feature integration supplements sparse keypoint information with visual context to improve recovered pose and shape accuracy.
  • Approach: WHAM estimates global trajectories using human motion and camera angular velocity, then refines root orientation and velocity using predicted foot-ground contact.The contact-aware refinement is designed to handle elevation changes and non-flat terrain such as stairs.
  • Results: 200 fps: WHAM’s network excluding preprocessing runs online and recursively predicts pose, shape, and global motion parameters.The method reports accurate camera trajectories and global body motions with minimal drift without global optimization.
  • Results: WHAM achieves state-of-the-art performance on 3DPW, RICH, and EMDB, outperforming image- and video-based methods on per-frame accuracy while maintaining temporal smoothness.The reported contributions include global-coordinate HPS regression and trajectory estimation for non-planar surfaces.

2. Related Work

Prior work includes image-based and video-based 3D human pose and shape estimation, but existing approaches face temporal inconsistency, limited video supervision, specialized hardware requirements, or expensive optimization. WHAM instead targets temporally coherent global motion from a standard monocular camera.

  • Image-Based 3D HPS Estimation: Image-based methods regress statistical body-model parameters from image features, using architectures such as HMR and often pseudo-ground-truth SMPL parameters.The related work discussion focuses on model-based direct regression rather than optimization-based methods.
  • Image-Based 3D HPS Estimation: Applying accurate image-based models independently to video frames can produce temporally inconsistent pose and shape, whereas WHAM aggregates temporal information for frame-accurate, temporally coherent estimation.The contrast motivates video-based modeling that preserves both per-frame accuracy and temporal consistency.
  • Video-Based 3D HPS: Video-based approaches use keypoints or plentiful 3D mocap data as proxies because video datasets with ground-truth SMPL poses are limited.Motion-lifting methods exploit mocap data to train networks that map 2D joints to 3D poses.
  • Global 3D Human Trajectory Estimation: Body-worn sensors are intrusive and require cooperation, while BodySLAM++ uses stereo cameras and an IMU; WHAM instead uses a standard monocular camera and does not require a camera gyro.WHAM can accept gyro input when available but treats it as optional.
  • Global 3D Human Trajectory Estimation: Optimization-based global trajectory methods combine motion priors with SLAM or scene information but are computationally expensive, while camera rotation can make trajectories noisy when camera cues are omitted.The related work contrasts these limitations with regression-based global trajectory estimation.

3. Methods

WHAM reconstructs pixel-aligned 3D human pose, shape, and motion from monocular video in world coordinates. It combines motion learned from synthetic AMASS data with visual features, camera motion, and contact-aware trajectory refinement for efficient online inference.

  • Network Architecture: WHAM encodes 2D keypoint sequences into motion features, integrates image features, and decodes camera-coordinate 3D motion with foot-ground contact.The motion decoder recovers SMPL parameters, weak-perspective camera translation, and contact probabilities from motion-feature history.
  • Network Architecture: The trajectory decoder combines motion features with camera angular velocity to estimate global root orientation and egocentric velocity.Camera angular velocity helps decouple human motion from camera motion and supports compatibility with SLAM or gyroscope measurements.
  • Contact Aware Trajectory Refinement: WHAM refines trajectory estimates using foot-ground contact to reduce foot sliding and generalize beyond flat surfaces, including stair climbing.The refinement adjusts egocentric root velocity using contact probability, then learns updates to root orientation and velocity before rolling out global translation.
  • Training: WHAM uses two-stage training: AMASS-based synthetic pretraining followed by video-based fine-tuning with feature integration and jointly optimized decoders.Synthetic pretraining teaches motion-context extraction and 3D lifting, while fine-tuning exposes the network to real keypoints and trains motion–image fusion.

4. Experiments

WHAM is evaluated for per-frame accuracy, temporal smoothness, global trajectory recovery, and component contributions across in-the-wild benchmarks. It achieves strong pose and trajectory results, with ablations showing benefits from feature integration, motion pretraining, camera angular velocity, and trajectory refinement.

  • Per-frame accuracy: WHAM (ViT) outperforms all previous methods on MPJPE, PA-MPJPE, and PVE across 3DPW, RICH, and EMDB.The comparison includes both per-frame and video-based methods; WHAM (Res) also outperforms every method except ReFit on RICH.
  • Inter-frame smoothness: WHAM has significantly lower acceleration error than state-of-the-art per-frame methods while preserving high per-frame accuracy.Compared with recent temporal methods, its acceleration error is comparable or slightly higher, whereas those methods tend to over-smooth motion.
  • Qualitative comparison: WHAM more accurately reconstructs the bending of the left knee during stair ascent than TCMR and GLoT.The qualitative comparison contrasts WHAM with methods that produce smooth results but fail to capture this motion detail.
  • Global trajectory recovery: WHAM outperforms existing methods on all global motion estimation metrics in the EMDB evaluation.The evaluation compares camera angular velocity from DPVO, DROID-SLAM, and ground-truth gyro measurements.
  • Global trajectory recovery: WHAM combined with DPVO is more accurate than integrating DPVO with HMR2.0, indicating active global-trajectory refinement.DROID-SLAM gives slightly better results than DPVO, while WHAM significantly outperforms TRACE on jitter and foot-sliding metrics.
  • Ablation study: Ablations show that feature integration, AMASS-based lifting pretraining, neural initialization, camera angular velocity, and trajectory refinement improve different aspects of performance.Removing camera angular velocity increases overall trajectory error, while removing trajectory refinement increases trajectory and foot-sliding errors but reduces jitter.

5. Conclusion

WHAM recovers accurate 3D human motion in global coordinates from monocular video with a moving camera, combining motion, visual, camera, and foot-contact information. The authors report strong benchmark performance and identify generalization and field-of-view assumptions as limitations.

  • 5. Conclusion: WHAM recursively lifts 2D keypoint sequences to 3D SMPL parameters and integrates temporal image context for pose and shape estimation.The method also uses camera angular velocity and foot-contact estimates to recover global motion.
  • 5. Conclusion: WHAM estimates global motion from a moving camera using camera angular velocity from SLAM or a camera gyro, together with foot-contact information.
  • 5. Conclusion: WHAM significantly outperforms existing image-based and video-based methods on in-the-wild benchmarks for 3D HPS and world-coordinate trajectory accuracy.
  • 5. Conclusion: AMASS-based motion learning limits generalization to motions outside the training distribution.The synthetic-data generation approach mainly assumes the full body remains within the field of view.
  • 5. Conclusion: Future work could use richer SLAM outputs for scene consistency or initialize optimization-based post-processing after online inference.

A. Synthetic data generation

WHAM addresses limited paired video and 3D-ground-truth data by synthesizing motion, keypoints, contact labels, and dynamic camera paths from AMASS. The generated data includes motion, shape, camera, and temporal variations for pretraining and training.

  • A. Synthetic data generation: WHAM pretrains on extensive synthetic 2D keypoint sequences generated from AMASS with ground-truth 3D poses.
  • A. Synthetic data generation: Training sequences contain L = 81 SMPL poses with motion-speed, root-rotation, and shape perturbations.Motion is sped up or slowed down by up to 50% of the original speed, with random root rotation and Gaussian shape noise.
  • A. Synthetic data generation: Contact labels are generated from foot velocity alone to support arbitrary ground conditions such as slopes and stairs.The contact probability uses threshold velocity v_t = 1cm/frame and coefficient α = 5.
  • A. Synthetic data generation: The virtual camera is initialized with sampled roll, pitch, and translation, while yaw is handled by random SMPL root rotation.Camera translation is sampled to keep the 3D pose within the field of view.
  • A. Synthetic data generation: Dynamic camera paths use interpolated extrinsics and 20% timestamp noise to model nonlinear camera motion.The synthesis process uses 6.7M frames sampled uniformly overall.

B. Datasets

The paper trains and evaluates WHAM using motion-capture, multiview, dynamic-camera, and in-the-wild datasets. These datasets provide complementary supervision for 3D pose, shape, and global trajectory estimation.

  • B. Datasets: Human3.6M contains indoor recordings of 15 actions from four calibrated cameras with motion-capture ground-truth 3D keypoints.Five subjects are used for training after downsampling to 25 fps.
  • B. Datasets: RICH is a large-scale multiview dataset spanning indoor and outdoor environments with ground-truth SMPL-X parameters.Its test split is used to evaluate 3D pose estimation accuracy.
  • B. Datasets: EMDB uses a dynamic camera and body-worn electromagnetic sensors, providing ground-truth SMPL parameters and global individual trajectories.

C. Losses

WHAM uses separate motion-reconstruction and trajectory-reconstruction losses to supervise body parameters, keypoints, camera motion, root trajectory, contact, and foot sliding. The losses combine direct ground-truth supervision with consistency and contact-aware terms.

  • C. Losses: WHAM groups its objectives into motion reconstruction and trajectory reconstruction losses.Motion terms include SMPL, vertices, 3D, 2D, and cascade losses; trajectory terms include root, contact, angular-velocity, camera, and foot-sliding losses.
  • C. Losses: When available, ground truth directly supervises SMPL parameters, vertices, root trajectory, and foot-contact probability with MSE losses.
  • C. Losses: 3D keypoint losses supervise both motion-encoder predictions and keypoints extracted from the predicted SMPL mesh.A cascade loss additionally penalizes their difference.
  • C. Losses: The 2D reprojection loss uses full-perspective projection when labels are available and weak-perspective projection for InstaVariety without full-frame images.
  • C. Losses: Camera-pose and angular-velocity losses are constructed from predicted root orientations in world and camera coordinates.The reconstructed camera angular velocity is derived from the camera rotation R.
  • C. Losses: Foot-sliding loss penalizes world-coordinate foot velocity during periods marked as foot contact.

D. Neural-network initialization

The section describes an initialization challenge in uni-directional RNNs and contrasts WHAM with a trajectory-refinement ablation.

  • D. Neural-network initialization: Uni-directional RNNs face different learning objectives for initial and subsequent frames because the initialization state is typically zero-padded.
  • D. Neural-network initialization: The first frame primarily relies on its input signal, whereas later frames also use information transferred from preceding frames.
  • D. Neural-network initialization: Figure 7 compares WHAM with a version without contact-aware trajectory refinement.

E. Qualitative evaluation of contact estimation

The qualitative evaluation shows that contact-aware trajectory refinement reduces foot sliding during contact, while the accompanying table concerns per-frame computation time.

  • E. Qualitative evaluation of contact estimation: Without trajectory refinement, WHAM induces significant foot sliding during contact.
  • E. Qualitative evaluation of contact estimation: With trajectory refinement, WHAM reconstructs a more feasible trajectory using its contact estimation.
  • E. Qualitative evaluation of contact estimation: Table 5 reports the per-frame computation time of each WHAM module in frames per second and milliseconds.

F. Run-time cost

WHAM’s core network is designed for online recursive prediction and runs substantially faster than prior methods, while full-system speed includes preprocessing costs.

  • F. Run-time cost: Around 9 fps: the full method with preprocessing runs on the EMDB dataset on a single A100 GPU.

G. Discussions of limitation

The discussion identifies failures in global motion estimation for unseen human-object interactions, non-foot contacts, and assumptions about full-body visibility.

  • G. Discussions of limitation: WHAM fails to capture global trajectories for people riding skateboards or bicycles because such scenarios are absent from its global-trajectory training data.
  • G. Discussions of limitation: Because contact estimation applies only to feet, WHAM can produce floating hands or sliding contact points for other body-ground contacts.
  • G. Discussions of limitation: The synthesis approach mainly assumes the full body remains within the field of view, although additional training augmentation could address this.
Loading 2312.07531v2…