PaperlayerView exact revision on arXiv

Source-linked AI summary

HuMoR: 3D Human Motion Model for Robust Pose Estimation

Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, Leonidas J. Guibas

arXiv:2105.04668v2cs.CVcs.LG

TL;DR

Recovering plausible 3D human motion from noisy and occluded observations remains difficult. HuMoR learns transition distributions with a conditional VAE and uses them in robust test-time optimization, outperforming strong baselines under heavy occlusion while estimating consistent ground planes and contacts.

  • Problem

    Existing methods struggle to recover plausible, varied 3D motion from noisy, partial, and occluded observations.

  • Method

    HuMoR is a conditional-VAE motion prior over pose transitions combined with test-time optimization of pose, shape, ground plane, and contacts.

  • Results

    HuMoR surpasses strong baselines in accuracy and physical plausibility under heavy occlusions while producing consistent ground-plane and contact outputs.

  • Takeaways & Limitations

    The model generalizes to novel motions and body shapes and supports reconstruction from 3D, RGB, and RGB-D observations.

  • Takeaways & Limitations

    HuMoR assumes a static camera and ground plane, limiting direct in-the-wild use with dynamic cameras or complex terrain.

Abstract

from arXiv · show

We introduce HuMoR: a 3D Human Motion Model for Robust Estimation of temporal pose and shape. Though substantial progress has been made in estimating 3D human motion and shape from dynamic observations, recovering plausible pose sequences in the presence of noise and occlusions remains a challenge. For this purpose, we propose an expressive generative model in the form of a conditional variational autoencoder, which learns a distribution of the change in pose at each step of a motion sequence. Furthermore, we introduce a flexible optimization-based approach that leverages HuMoR as a motion prior to robustly estimate plausible pose and shape from ambiguous observations. Through extensive evaluations, we demonstrate that our model generalizes to diverse motions and body shapes after training on a large motion capture dataset, and enables motion reconstruction from multiple input modalities including 3D keypoints and RGB(-D) videos.

1. Introduction

HuMoR addresses robust 3D human motion estimation under noisy, partial, and occluded observations by modeling pose transitions and using them in test-time optimization.

  • Existing methods struggle with varied motions, body shapes, modalities, noisy or oversmoothed motion, footskate, and occlusions.
  • Pose-based models limit environment-interaction modeling or realism, while physical dynamics models are resource intensive and require unobservable quantities.
  • HuMoR models a probability distribution over pose transitions with a conditional variational autoencoder rather than directly modeling likely poses.
  • The model interprets latent transitions as generalized forces, decodes them through numerical integration, and explicitly predicts ground contacts for estimation constraints.
  • HuMoR combines motion-prior modeling with robust test-time optimization over pose, shape, ground plane, and contact points from RGB(-D), 2D, or 3D inputs.
  • Comprehensive evaluations report higher accuracy and physical plausibility than state-of-the-art methods under partial and severe occlusions, with generalization to diverse motions and body shapes.

2. Related Work

Prior work estimates pose, shape, and motion through learned regressors, optimization, pose-space models, and physics-based approaches, but each has important scope or realism limitations.

  • Learning-based methods regress pose and shape from images or videos, often producing accurate root-relative joints but using fixed-length or recurrent motion representations.
  • Optimization-based methods fit body models to observations and commonly use pose, shape, or temporal smoothness priors.
  • Interaction and contact constraints can improve estimation when scene geometry is known, whereas HuMoR jointly uses an expressive motion model for natural motion and ground contact.
  • Early human-motion models operate in pose space and are limited to specific motions, while physics-based models may generalize more accurately and estimate environmental information.
  • Animation models often target visual fidelity for small character sets or periodic locomotion, and broader motion and shape models have focused on short-term prediction.
  • HuMoR is closest to Motion VAE but adds contributions intended to generalize to unseen, non-periodic motions and novel body shapes.

3. HuMoR: 3D Human Dynamics Model

HuMoR is a transition-based generative dynamics model whose conditional VAE supports autoregressive motion generation and robust optimization over observed motion variables.

  • State Representation: The state includes root translation and orientation, joint angles and positions, plus their velocities, while SMPL maps pose and shape parameters to mesh vertices and joints.
  • Latent Variable Dynamics Model: HuMoR models each transition conditionally on the previous state with a latent variable representing the possible change to the current state.
  • Latent Variable Dynamics Model: The decoder outputs an additive state change and contact probabilities; the contact predictions provide environmental constraints for TestOpt.
  • Rollout: At inference, alternating samples from the conditional prior and decoder generates sequences whose latent transitions act like forces and whose decoder performs dynamics integration.
  • Rollout: A deterministic rollout maps an initial state and latent-transition sequence to a resulting state through autoregressive integration of additive updates.
  • Training: Training uses reconstruction, KL, SMPL, consistency, and contact losses, with a separately trained 12-component initial-state GMM and canonical coordinate frames.

4. Test-time Motion Optimization

TestOpt uses HuMoR to recover motion and shape from noisy, partial observations by optimizing latent motion variables alongside scene and body parameters. Its MAP objective combines modality-specific data fitting with motion-prior and plausibility regularization.

  • TestOpt parameterizes motion with the initial state and latent transitions, while jointly optimizing ground plane and body shape.The observation frame is related to HuMoR’s canonical frame through the optimized ground plane.
  • The MAP objective seeks motion that is plausible under HuMoR while closely matching the observations.The objective combines a generative-model prior with data and regularization energies.
  • The data term adapts to 3D joints, 2D joints, or 3D point clouds, using body-model joints or mesh vertices.For 2D observations, reprojection uses detection confidence and a robust Geman-McClure loss; segmentation can suppress spurious joints.
  • Four regularizers enforce skeleton consistency, environment consistency, ground proximity, and shape plausibility during optimization.The environment term links predicted contacts, motion, and the scene; contact height keeps contacting joints near the floor.
  • Optimization initializes latent variables through CVAE inference and uses L-BFGS with autograd for fitting.With batching, fitting a 3s RGB video takes about 5.5 min.

5. Experimental Results

HuMoR is evaluated as both a generative motion model and a motion prior for TestOpt across synthetic and real-world observations. Results cover future prediction, sampling, partial or noisy 3D data, and RGB(-D) inputs, emphasizing accuracy, plausibility, generalization, and occlusion handling.

  • Evaluation Setup: HuMoR is evaluated on generative sampling and TestOpt-based motion estimation from 3D, RGB, and RGB-D inputs.Experiments use AMASS, i3DB, and PROX, including held-out motions and body shapes.
  • Generative Model Evaluation: The generative evaluation measures future prediction on 2s sequences using contact accuracy, ADE, and FDE, and diversity on 5s rollouts using APD.Fifty samples are generated for each initial state or rollout evaluation, with the lowest-ADE sample used for prediction.
  • Generative Model Evaluation: HuMoR’s CVAE improves both sample accuracy and diversity over MVAE, although HuMoR (Qual) trades prediction accuracy for better diversity and visual quality.The qualitative variant enforces skeleton consistency by using JSMPL during rollout but is too expensive for TestOpt.
  • Qualitative Results: Qualitative results show plausible non-periodic motions from partial 3D keypoints and natural sitting and walking under heavy RGB-video occlusions.HuMoR also recovers highly dynamic motions such as dancing when the full body is visible.
  • Estimation from 3D Observations: For partial or noisy 3D observations, TestOpt with HuMoR gives more accurate occluded poses, smoother motion, fewer severe penetrations, and 97% person-ground contact accuracy under severe noise.The noisy-joint setting adds Gaussian noise with 4cm standard deviation; VPoser-t is smoother but less accurate because it oversmooths motion.
  • Estimation from RGB(-D) Observations: On real RGB video with occlusions, HuMoR outperforms baselines on positional and plausibility measures, while the conditional prior and LSMPL have the largest ablation impact.Even without contact predictions, the method outperforms the baselines; RGB-D further improves performance and slightly surpasses PROX-D.

6. Discussion

HuMoR is a learned generative 3D human-motion model used during test-time optimization to recover plausible pose and shape from diverse observations. It performs well under heavy occlusion while producing consistent ground-plane and contact outputs, but assumes static cameras and ground planes and uses a simplistic contact model.

  • HuMoR models 3D human motion generatively and is leveraged during test-time optimization to recover pose and shape from 3D, RGB, and RGB-D observations.
  • The method generalizes to novel motions and body shapes for both generative tasks and downstream optimization.
  • HuMoR excels at estimating plausible motion under heavy occlusions while simultaneously producing consistent ground-plane and contact outputs.
  • The current formulation assumes a static camera and ground plane and uses a simplistic contact model, limiting direct in-the-wild operation and scene-person interaction modeling.

Appendices

The appendices expand HuMoR’s state representation, model design, optimization rationale, evaluation metrics, and implementation details, while documenting limitations and supplementary results.

  • Appendix overview: The appendices provide extended discussions, HuMoR and TestOpt method details, a probabilistic optimization derivation, experimental details, and additional evaluations.Supplementary videos and qualitative results are also referenced on the project webpages.
  • State representation: HuMoR redundantly represents explicit joint positions and SMPL parameters, using extrinsic keypoints to help learn motion characteristics such as static contact.The state does not explicitly encode full-body shape; only bone proportions are implicitly represented through joint locations.
  • Model design: Compared with MVAE, HuMoR adds a learned conditional prior, predicts state changes, models person-ground contacts, and uses SMPL regularization.HuMoR also places the root at the pelvis and represents full 3D velocities, unlike MVAE’s ground-projected root and lower-dimensional velocities.
  • Optimization: Additional skeleton and environment regularizers help optimization when learned-prior likelihoods are unreliable outside the clean AMASS training distribution.Without regularization, the motion prior can encounter local minima and converge slowly from initial states far from the training domain.
  • Evaluation: Evaluation uses global joint accuracy, root-aligned local accuracy, and motion plausibility because traditional positional metrics omit global motion and physical or perceptual plausibility.The appendix notes that these metrics still have shortcomings and that more informative motion-estimation metrics are needed.
  • Optimization: TestOpt reports only a locally optimal solution because it uses L-BFGS on a mixture of convex and non-convex objectives.Global-search methods such as simulated annealing or MCMC/HMC are described as computationally prohibitive in this setting.

A.1. Assumptions and Limitations

The method relies on ground-plane and static-camera assumptions, and has failure modes under severe occlusion, uncommon motions, and near-static observations.

  • Ground-plane assumption: In-the-wild operation requires a consistent reference frame obtainable from an accelerometer, static objects, fiducial tags, or gravity-direction estimates.The ground plane is used to transform observations into the canonical frame in which the prior was trained.
  • Ground-plane assumption: The CVAE is trained with a ground plane at z = 0, gravity in the −z direction, and no complex terrain interactions.The authors note that this training assumption, rather than TestOpt itself, makes the ground plane necessary.
  • Static-camera assumption: All evaluations assume a static camera, although structure from motion, SLAM, or camera relocalization could support moving-camera scenarios.Disambiguating camera and human motion with the learned model remains future work.
  • Optimization limitations: Optimization is slower than direct prediction and currently returns only a single local optimum rather than multiple hypotheses for uncertainty characterization.The authors identify learned optimization and multi-hypothesis outputs as future directions.
  • Failure cases: Extreme first-frame occlusions, uncommon motions, and nearly static people can produce local minima, spurious ground planes, or implausible poses.The cited examples include laying down and ambiguity between standing and sitting.

B.1. CVAE Architecture and Implementation

HuMoR uses canonicalized motion states and MLP-based CVAE components, trained on AMASS with strategies addressing redundancy, posterior collapse, and rollout robustness.

  • Representation: HuMoR and TestOpt use SMPL+H but model only the 22 body joints, excluding hand joints from the current scope.Hand joints could be optimized alongside body motion but are not modeled here.
  • Representation: Inputs are transformed into a canonical frame by removing root x/y translation and aligning the body’s right axis with +x.This normalization is intended to ease learning and improve generalization.
  • Architecture: The encoder and prior use five-layer MLPs, while the decoder uses four layers; the 48-dimensional latent transition is skip-connected throughout the decoder.ReLU activations and 16-group normalization are used between non-output layers.
  • Training: Training uses AMASS, a standardized database with over 11k motion sequences from 344 people, with separate training, validation, and held-out evaluation subsets.The listed evaluation subsets are HumanEva and Transitions.
  • Training: KL annealing, scheduled sampling, and staged supervision with self-rollouts address posterior collapse and long-term generation robustness.The KL weight is annealed during early training, followed by mixed true/self inputs and full self-rollouts.
  • Test-time implementation: Test-time optimization represents pose with VPoser latents, uses SMPL-generated joints, and combines motion, data, ground, shape, and environment-related terms.The motion prior receives higher weighting for heavily occluded i3DB data than for more visible PROX RGB data.

D. MAP Objective Derivation

The MAP derivation interprets TestOpt as maximizing the posterior over initial state, latent transitions, ground, and shape, yielding the paper’s energy-minimization objective.

  • MAP formulation: TestOpt seeks a MAP estimate over the initial state x0, latent transitions z1:T, ground g, and shape β given observations y0:T.The optimization variables and observation sequence are explicitly defined in the probabilistic formulation.
  • Energy objective: The posterior objective becomes a log-likelihood consisting of data and prior terms over the optimized variables.The derivation writes the objective as log p(y0:T | x0, z1:T, g, β) + log p(x0, z1:T, g, β).
  • Energy objective: The resulting minimization combines motion, data, ground, and shape energies: Emot + Edata + Egnd + Eshape.An earlier equivalent form includes ECVAE and Einit before they are consolidated into the motion term.
  • Prior terms: The derivation recovers the main optimization objective under probabilistic energy interpretations, with additional skeletal and environment regularizers.Those extra regularizers could in principle be incorporated into a more complex motion prior.
  • MAP formulation: The likelihood term is decomposed according to the observation model, with later observations depending on the initial state and preceding CVAE transitions.The rollout relation is represented as xt = f(x0, z1:t).
  • Prior terms: The ground prior keeps g near initialization, while the shape prior keeps β near the neutral zero vector.Both are represented as Gaussian priors and corresponding quadratic energies.

E.1. Datasets

The evaluation uses held-out motion datasets and RGB/RGB-D interaction datasets, alongside motion and estimation baselines with modality-specific metrics and preprocessing.

  • Datasets: HumanEva and Transitions are held-out evaluation subsets containing four subjects and approximately 19 minutes of motion.These subsets are used for the main paper’s experiments in Sections 5.3 and 5.4.
  • Datasets: i3DB contains 30 Hz RGB videos with medium-to-heavy occlusions, annotated 3D joints at 10 Hz, and primitive cuboid scene reconstructions.OpenPose, person segmentation, and plane detection provide inputs and initialization.
  • Datasets: PROX contains 30 Hz RGB-D person-scene videos with light-to-medium occlusions; the selected scenes include 12 people and about 19 minutes of video.Because pose annotations are unavailable, evaluation uses scanned scene meshes for ground-plane plausibility.
  • Baselines: The MVAE baseline applies all listed HuMoR ablations simultaneously, while retaining the same training procedure and test-time energy weightings where applicable.Removed components include delta prediction, contact prediction, SMPL losses, and the learned conditional prior.
  • Baselines: VPoser-t is the initialization-only baseline, while PROX-RGB, PROX-D, and VIBE use distinct body-fitting or inference setups.VIBE uses the same OpenPose detections as the proposed method for a fairer comparison under occlusion.
  • Metrics: Evaluation reports visible and occluded positional errors, acceleration magnitude, and ground-penetration frequency using modality- and dataset-specific procedures.For i3DB, segmentation masks determine visibility and errors use a 12-joint subset.

F.1. Qualitative Evaluation

Qualitative evaluations show that TestOpt with HuMoR handles occlusions, noise, contacts, scene interaction, and highly dynamic motions across multiple input modalities.

  • Occluded 3D keypoints: TestOpt with HuMoR produces plausible lower-body motion from occluded 3D keypoints, unlike VPoser-t and MVAE baselines.VPoser-t fails to produce plausible lower-body motion, while MVAE often misaligns with observations.
  • Noisy 3D joints: HuMoR recovers complex crawling contacts involving feet, hands, and knees from noisy 3D joints.The estimated motion includes contact patterns beyond foot-ground interactions.
  • RGB video: On PROX RGB videos, HuMoR maintains temporally coherent motion under occlusion where PROX-RGB often reverts to a mean leg pose.PROX-RGB benefits from scene-mesh input for fully visible people but remains weak under occlusions.
  • RGB-D video: On PROX RGB-D videos, HuMoR produces natural motion within scenes despite not using the scene mesh as input.The motion prior supports plausible interaction with the environment.
  • Dynamic motion: HuMoR generalizes to difficult dance motions from full-body 3D keypoints despite limited dancing data during training.The model captures these motions at test time because it operates on pairs of frames; related results use 2D joints from AIST videos.

F.3. Sensitivity to Occlusions and Noise

Sensitivity experiments evaluate TestOpt with HuMoR under increasing occlusion and observation noise, showing stable performance as observations become less reliable.

  • Occlusions: HuMoR outperforms MVAE without occlusions and remains comparable to MVAE as occlusions increase, while VPoser-t degrades greatly.With clean points, VPoser-t fits most closely; increasing occlusion shifts reliance toward the motion prior.
  • Occlusions: After knees become occluded, occluded-keypoint errors saturate because performance depends nearly entirely on the motion prior.Occlusion thresholds range from 0.0 to 1.2 m, progressively hiding the lower body.
  • Qualitative robustness: TestOpt with HuMoR recovers complex contact patterns involving feet, knees, and hands from noisy 3D joint observations.The crawling example includes both estimated motion and contacts.
  • Generalization: HuMoR generalizes to dynamic dance motions and can fit inaccurate 2D joint detections in AIST videos.Examples include DanceDB motions and highly dynamic cartwheel sequences.
Loading 2105.04668v2…