PaperlayerView exact revision on arXiv

Source-linked AI summary

GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras

Ye Yuan, Umar Iqbal, Pavlo Molchanov, Kris Kitani, Jan Kautz

arXiv:2112.01524v2cs.CVcs.AIcs.GRcs.LGcs.RO

TL;DR

GLAMR addresses global human mesh recovery from monocular videos with dynamic cameras, where long-term occlusions and joint camera-motion ambiguity are central challenges. It infills missing motion, predicts global trajectories, and optimizes trajectories and camera poses, achieving state-of-the-art results on challenging datasets while retaining several practical limitations.

  • Problem

    Dynamic cameras and severe, long-term occlusions make consistent global-coordinate human mesh recovery difficult, especially when people leave the camera’s field of view.

  • Method

    GLAMR uses a generative motion infiller, a global trajectory predictor based on local body motion, and global optimization of trajectories and camera parameters against video evidence.

  • Results

    GLAMR achieves state-of-the-art results and outperforms prior methods on motion infilling and global human mesh recovery across challenging indoor and in-the-wild datasets.

  • Takeaways & Limitations

    The approach provides global-coordinate human mesh recovery for monocular videos recorded with dynamic cameras, including cases with severe occlusion and people outside the camera’s field of view.

  • Takeaways & Limitations

    Sequential dependence across five stages can propagate early errors to later stages and lead to inaccurate global pose estimation.

Abstract

from arXiv · show

We present an approach for 3D global human mesh recovery from monocular videos recorded with dynamic cameras. Our approach is robust to severe and long-term occlusions and tracks human bodies even when they go outside the camera's field of view. To achieve this, we first propose a deep generative motion infiller, which autoregressively infills the body motions of occluded humans based on visible motions. Additionally, in contrast to prior work, our approach reconstructs human meshes in consistent global coordinates even with dynamic cameras. Since the joint reconstruction of human motions and camera poses is underconstrained, we propose a global trajectory predictor that generates global human trajectories based on local body movements. Using the predicted trajectories as anchors, we present a global optimization framework that refines the predicted trajectories and optimizes the camera poses to match the video evidence such as 2D keypoints. Experiments on challenging indoor and in-the-wild datasets with dynamic cameras demonstrate that the proposed approach outperforms prior methods significantly in terms of motion infilling and global mesh recovery.

1. Introduction

GLAMR addresses global human mesh recovery from monocular videos with dynamic cameras, where consistent global coordinates and severe, long-term occlusions challenge existing methods. It combines motion infilling, global trajectory prediction, and optimization, achieving stronger performance than prior methods on challenging datasets.

  • Challenges: Dynamic cameras complicate consistent global-coordinate estimation, while SLAM can fail in-the-wild and introduce scale ambiguity.Existing methods commonly estimate camera- or root-relative meshes and require SLAM to recover global meshes from dynamic-camera videos.
  • Challenges: Severe and long-term occlusions arise from missed detections, complete obstruction, or people leaving the camera’s field of view.These conditions undermine methods that depend on detections or visible body parts.
  • Approach: GLAMR autoregressively infills occluded people’s local body motions with a deep generative motion infiller based on visible motions.The method uses motion dynamics learned from human movement to address potentially severe occlusions.
  • Approach: GLAMR predicts global human trajectories from local body motions and uses them as anchors for global motion and camera optimization.This design addresses ambiguity in jointly reconstructing human motions and camera poses.
  • Results: Experiments on challenging indoor and in-the-wild datasets show significant improvements over prior state-of-the-art methods for occlusion handling and global mesh estimation.The reported evaluation covers both motion infilling and global human mesh recovery.

2. Related Work

Prior work largely estimates camera- or root-relative poses, addresses only partial occlusions, or assumes static and multi-view capture settings. GLAMR instead targets monocular dynamic-camera videos with long-term occlusions and global-coordinate reconstruction.

  • Camera-Relative Pose Estimation: Camera-relative methods address depth ambiguity by estimating meshes relative to the camera or pelvis, often neglecting absolute 3D translation.Some approaches recover translation through optimization, scene constraints, physics-based constraints, or 2.5D representations.
  • Global Pose Estimation: Most global-pose methods assume calibrated, synchronized, static multi-view capture, while alternatives may require audio, IMUs, or pre-scanned environments.These assumptions differ from GLAMR’s monocular dynamic-camera setting.
  • GLAMR: GLAMR’s framework preprocesses tracked people, infills occluded motion, predicts global trajectories, and jointly optimizes trajectories with camera parameters against video evidence.The overview identifies four stages, including camera-aware global optimization.
  • Occlusion-Aware Pose Estimation: Existing occlusion-aware pose methods generally handle partial occlusions but not complete obstruction or people remaining outside the camera’s field of view for long periods.GLAMR is designed for severe and long-term occlusions.
  • Human Motion Modeling: Motion infilling research generates complete motions from partial observations, while related work also predicts global translations from local joint positions.GLAMR combines these lines of work in a global mesh-recovery framework.

3. Method

GLAMR recovers occlusion-free human motions and global meshes from dynamic-camera videos through generative infilling, trajectory prediction, and joint camera-motion optimization.

  • Framework: The framework takes dynamic-camera video and represents each person’s global motion with root translations, rotations, body poses, and shapes.SMPL maps each global pose to a mesh sequence.
  • Generative Motion Infilling: A generative motion infiller reconstructs complete body motion from partially observed poses and a visibility mask using a CVAE.Different latent codes can produce different plausible occlusion-free motions.
  • Generative Motion Infilling: At test time, autoregressive sliding-window infilling uses already completed context and look-ahead visible frames to fill intervening motion.The look-ahead guides ending motion and helps avoid discontinuities, while windows advance by h_o frames.
  • Generative Motion Infilling: The motion-infilling network uses a Transformer encoder-decoder that conditions generated motion on visible poses and a latent code.Training synthesizes occlusions in AMASS motions and uses a CVAE objective with reconstruction and KL terms.
  • Global Trajectory Prediction: A CVAE-based global trajectory predictor converts infilled body motions into global root translations and rotations through an egocentric trajectory representation.LSTMs model local trajectory changes because they depend mainly on nearby body motion rather than long-range temporal context.
  • Global Optimization: Global optimization jointly refines all people’s trajectories and per-frame camera extrinsics to match video evidence such as 2D keypoints.The optimized output combines the predicted global trajectories with the occlusion-free body motions and shapes from earlier stages.

4. Experiments

GLAMR is evaluated on AMASS, Dynamic Human3.6M, and 3DPW using global, occlusion-aware metrics and comparisons against motion-infilling and camera-estimation baselines. It outperforms competing approaches in global reconstruction and motion infilling, while ablations support the value of its trajectory prediction and egocentric trajectory representation.

  • Datasets: Experiments use AMASS for motion-infiller and trajectory-predictor evaluation, 3DPW for in-the-wild testing, and Dynamic Human3.6M for simulated dynamic cameras and occlusions.Dynamic Human3.6M is created by cropping Human3.6M frames with an oscillating small view window.
  • Evaluation protocol: Global evaluation uses G-MPJPE and G-PVE, computed with 10-second sliding-window alignment because dynamic-camera trajectory errors accumulate over time.3DPW lacks accurate ground-truth global trajectories, so these metrics are not reported there.
  • Overall results: GLAMR outperforms baselines in almost all metrics, including lower global errors and lower FID and PA-MPJPE for occluded poses.Its visible-pose PA-MPJPE is nearly tied with the best method, while its acceleration error is smaller.
  • Motion infilling: GLAMR’s generative motion infiller achieves better PA-MPJPE for sampled and reconstructed infilled motions and halves ConvAE’s FID on AMASS.The lower FID indicates that its infilled motions are closer to real human motions.
  • Trajectory prediction: The trajectory predictor outperforms Transformer and direct 6-DoF global-trajectory variants on best-of-five G-MPJPE and G-PVE.The authors attribute this to LSTMs’ longer-motion generalization and the use of egocentric rather than large global offsets.
  • Global optimization: Removing either the trajectory predictor or egocentric trajectory representation substantially worsens global reconstruction, increasing G-MPJPE, G-PVE, and acceleration error.These ablations support both components as important to the global optimization framework.

5. Discussion and Limitations

GLAMR combines motion infilling, trajectory prediction, and global optimization to recover human meshes in consistent global coordinates from dynamic-camera videos. The paper reports state-of-the-art results while identifying several practical limitations.

  • Discussion: The method targets 3D human mesh recovery in consistent global coordinates from videos captured by dynamic cameras.Its motion infiller addresses severe occlusions, while trajectory prediction resolves ambiguity in jointly reconstructing global motions and camera poses.
  • Discussion: GLAMR predicts global human trajectories from local body motions and uses them as anchors for camera optimization.The global optimization framework refines the predicted trajectories and optimizes camera poses.
  • Discussion: GLAMR achieves state-of-the-art results on challenging datasets and advances global human mesh recovery in the wild.The reported results cover the paper’s target recovery problem rather than a single secondary metric.
  • Limitations: The authors identify error propagation across multiple stages, limited body shape estimation, non-real-time processing, and absent scene information as limitations.These limitations are described as avenues for future work.

B. Implementation Details for Preprocessing

The preprocessing pipeline uses multi-object tracking and re-identification alongside initial human pose and shape estimators, while evaluation uses ground-truth tracks.

  • Preprocessing: DeepSORT with ResNet-50 in MMTracking provides 3D multi-object tracking and re-identification.The implementation uses DeepSORT [103] with ResNet-50 [29] in the MMTracking package.
  • Preprocessing: Ground-truth tracks are used to evaluate GLAMR and the baselines following the standard human pose estimation protocol.This separates the reported evaluation from errors in predicted tracking.
  • Preprocessing: KAMA or SPEC provides the initial human pose and shape estimates from bounding boxes extracted for tracked people.These estimators supply the initial camera-coordinate pose and shape sequences used downstream.

C. Implementation Details for Generative Motion Infiller

The CVAE-based generative motion infiller uses a Transformer architecture and autoregressive sliding-window training to synthesize motions across long occlusions.

  • Architecture: The motion infiller uses Transformer modules with 256-dimensional keys, queries, and values, two blocks, 512-dimensional feedforwards, dropout 0.1, and eight attention heads.Token-wise MLPs use hidden layers of 512 and 256 with ReLU activations.
  • Architecture: The posterior network appends two learnable tokens to the ground-truth pose sequence to output posterior latent-code parameters.The cited architecture passages describe the posterior mean output for latent code z.
  • Training: The infiller uses a 128-dimensional latent code and autoregressive windows of 50 frames with 10 context and 10 look-ahead frames.Occluded spans during training contain 10 to 40 consecutive frames, excluding the first 10 context frames.

D. Implementation Details for Global Trajectory Predictor

The global trajectory predictor represents heading and motion using body-centered features, then models trajectory distributions with bidirectional LSTMs, MLPs, pooling, and a CVAE objective.

  • Representation: The heading vector is obtained by aligning the person’s root z-axis with the world z-axis and taking the aligned root y-axis.This representation is reported as more stable than Euler-angle yaw because yaw suffers from singularities.
  • Architecture: The trajectory predictor uses two bidirectional LSTM layers with hidden dimension 256 and token-wise MLPs with 512- and 256-dimensional hidden layers.The architecture is described for the CVAE-based global trajectory predictor.
  • Architecture: Token-wise mean pooling summarizes prior and posterior token sequences into single features for latent-distribution parameters.The trajectory predictor uses this pooling operation in both networks.
  • Representation: Input poses are converted to 3D joint positions without global rotations and translations before prediction.The implementation reports better performance with 3D joint positions than with joint rotations directly.
  • Training: The trajectory predictor uses a 128-dimensional latent code and trains with a CVAE objective whose KL term is weighted by 0.001.Training runs for 2000 epochs with batch size 256 and a learning rate of 0.0001.

E. Implementation Details for Global Optimization

Global optimization is initialized from predicted human trajectories and approximate camera intrinsics, then refined with fixed optimization settings. A one-minute scene takes around five minutes to process.

  • Initialization: Camera intrinsics are approximated from image dimensions with the principal point at the image center and remain fixed during optimization.
  • Initialization: Camera extrinsics are initialized from persons’ predicted global trajectories, while frames without visible people reuse the most recent visible-frame extrinsics.
  • Hyperparameters and Optimization: The optimization coefficients are dataset-specific, and the inter-person penetration loss is disabled for Human3.6M because each video contains one person.
  • Computation Time: A one-minute scene takes around 5 minutes with 500 optimization iterations, compared with more than 30 minutes using OpenSfM.

F. Evaluation of Global Optimization on 3DPW

Global optimization improves the spatial relationships between humans, while sliding-window length exposes a trade-off between reconstruction and sample quality. With no visible pose, the motion infiller can still generate plausible motions from its learned prior.

  • Global Optimization: Global optimization greatly reduces relative translation and rotation errors between humans on 3DPW.Relative translation is measured in meters and relative rotation in angles between pairs of humans.
  • Sliding Window Length: Increasing window length h increases reconstruction error because the latent code must represent more motion variations.The context and look-ahead lengths are each set to 0.2h.
  • Sliding Window Length: Sample error first decreases and then increases with window length, reflecting a trade-off between added context and greater latent-code burden.
  • Motion Infilling without Visible Pose: With no visible pose, the motion infiller can still produce plausible motions sampled from its training prior as an unconditional VAE.

H. Discussion of Limitations

The paper identifies sequential error propagation, limited mesh detail, non-real-time processing, and independent person modeling as method limitations. It also flags privacy and misinformation risks from misuse of reconstructed motion.

  • Method Limitations: The five sequential stages can propagate early errors into inaccurate late-stage global pose estimates.The authors suggest integrating the stages into an end-to-end learnable framework.
  • Method Limitations: The method recovers SMPL parameters, which omit fine details such as clothing.Neural articulated shapes are suggested as a possible remedy.
  • Method Limitations: Batch processing and global optimization prevent real-time operation, motivating future causal optimization over a small incoming-frame window.
  • Method Limitations: Per-person motion infilling and trajectory prediction may miss complex interactions between people.
  • Potential Negative Impact: Misuse with neural rendering could fabricate human-action videos indistinguishable from real ones, raising privacy and misinformation concerns.The paper calls for research on detecting synthesized videos with realistic human motion.
Loading 2112.01524v2…