PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Human and Camera Motion from Videos in the Wild

Vickie Ye, Georgios Pavlakos, Jitendra Malik, Angjoo Kanazawa

arXiv:2302.12827v2cs.CV

TL;DR

In-the-wild videos make global human motion ambiguous because camera and human movement are entangled, while existing alternatives often require camera-aware capture or dense scene reconstruction. The paper jointly optimizes relative camera motion, tracked human trajectories, and learned motion priors to recover people in a shared world frame. It reports improved performance over prior methods on EgoBody and improved downstream tracking on PoseTrack, while noting failures in ill-posed multiview geometries.

  • Problem

    Global human trajectories are difficult to recover from in-the-wild videos because image motion combines human and camera motion, while alternatives often require extra sensors or dense scene reconstruction.

  • Method

    The method jointly optimizes relative camera estimates, tracked human trajectories, and learned human-motion priors to place multiple people in a shared world coordinate frame.

  • Results

    The approach outperforms state-of-the-art methods on EgoBody and improves downstream PHALP tracking on PoseTrack using recovered scaled camera motion.

  • Takeaways & Limitations

    Recovering camera scale enables global reasoning about multiple people and supports more persistent 3D registration for tracking in challenging videos.

  • Takeaways & Limitations

    The method can recover inconsistent trajectories in in-the-wild videos with predominantly rotational camera motion or co-linear human-camera motion.

Abstract

from arXiv · show

We propose a method to reconstruct global human trajectories from videos in the wild. Our optimization method decouples the camera and human motion, which allows us to place people in the same world coordinate frame. Most existing methods do not model the camera motion; methods that rely on the background pixels to infer 3D human motion usually require a full scene reconstruction, which is often not possible for in-the-wild videos. However, even when existing SLAM systems cannot recover accurate scene reconstructions, the background pixel motion still provides enough signal to constrain the camera motion. We show that relative camera estimates along with data-driven human motion priors can resolve the scene scale ambiguity and recover global human trajectories. Our method robustly recovers the global 3D trajectories of people in challenging in-the-wild videos, such as PoseTrack. We quantify our improvement over existing methods on 3D human dataset Egobody. We further demonstrate that our recovered camera scale allows us to reason about motion of multiple people in a shared coordinate frame, which improves performance of downstream tracking in PoseTrack. Code and video results can be found at https://vye16.github.io/slahmr.

1. Introduction

The paper targets global human trajectory recovery when camera motion obscures world motion in in-the-wild video. It combines relative camera estimates, human tracking, and learned motion priors in a joint optimization evaluated on EgoBody and PoseTrack.

  • Motivation: 2D person motion alone cannot recover world trajectories when a moving camera follows the person, so camera motion must also be estimated.The camera and human trajectories project jointly into image motion, obscuring their individual contributions.
  • Contribution: The method models camera motion to reconstruct 3D human motion and multiple people in a shared world coordinate frame.This shared frame captures spatial relationships between people.
  • Motivation: Global trajectory methods commonly require extra sensors or dense scene reconstruction, whereas this approach targets videos in the wild without capture-setup or environment constraints.The approach is designed for dynamic cameras, including large-scale egocentric video datasets.
  • Method: The pipeline estimates relative camera motion from static-scene pixel motion, tracks people and poses, then jointly optimizes world trajectories using video observations and learned human-motion priors.These estimates initialize human and camera trajectories in a shared world frame.
  • Method: Relative camera motion remains useful despite degenerate scene reconstruction, while realistic human-motion priors and multiple people constrain the camera-displacement scale.The method uses human displacement to resolve the scale of relative camera estimates.
  • Evaluation: The method improves over a state-of-the-art approach on EgoBody and uses recovered scaled camera motion to improve downstream PHALP tracking on PoseTrack.PoseTrack has no 3D ground-truth labels, so the paper evaluates the recovered camera through tracking.

2. Related Work

Prior work often recovers people in camera coordinates or local motion, while global trajectories typically depend on known camera geometry, dense scene reconstruction, or controlled capture. SLAHMR instead combines tracked human representations with camera-aware world-frame optimization.

  • Human Mesh Recovery from a Single Image: Single-image mesh-recovery methods generally predict people in camera coordinates, while PHALP tracks identities and 3D meshes frame by frame.The paper uses PHALP detections and predicted meshes to initialize its camera-aware reconstruction.
  • Human Mesh Recovery from Video: Video mesh-recovery methods often model local motion without camera motion, and local-pose-only approaches can produce unreliable global trajectories under occlusion or unusual poses.The paper explicitly accounts for camera motion to place people in the static scene.
  • Optimization-Based Methods: Optimization methods using smoothness or learned dynamics priors have generally been limited to settings with metrically known cameras because the priors are defined in human coordinates.This leaves camera-aware global reconstruction as an unresolved setting for those approaches.
  • Scene-Based Methods: Methods relying on prior 3D scene information can recover global trajectories, but dense reconstruction requires many viewpoints and wide baselines, often from controlled or television capture.Such capture conditions differ from unconstrained in-the-wild video.
  • Multiple People: Multi-person reconstruction has been studied with optimization, regression, and consistency constraints, but these works do not establish the paper’s camera-aware global trajectory setting.The paper’s pipeline uses identities and local poses for multiple people before world-frame optimization.

3. Method

The method reconstructs people and camera motion in a shared world frame by combining camera estimates from scene pixel motion with human motion priors. Joint optimization uses image reprojection, trajectory smoothing, learned motion likelihoods, and environmental contact constraints to resolve camera scale and produce plausible global trajectories.

  • Representation: The system represents each person with SMPL-H pose, shape, global orientation, and root translation parameters over time.These parameters generate the person’s mesh vertices and joints in the world frame.
  • Initialization: PHALP supplies per-frame poses and identity associations in camera coordinates, where observed motion combines human and camera motion.The method initializes world-frame people and cameras using these estimates together with SLAM transforms.
  • Initialization: SLAM estimates relative world-to-camera transforms from static-scene pixel motion, while optimization jointly determines camera scale and global human trajectories.The camera estimate is scale-ambiguous, so the method uses human motion constraints to place people in the world.
  • Optimization: A reprojection objective aligns world-frame joints with detected 2D keypoints using perspective projection and a robust Geman-McClure loss.The initial stage optimizes global orientation and root translation because reprojection alone is under-constrained.
  • Optimization: Trajectory smoothing and learned HuMoR transition priors constrain world motion to be temporally smooth and physically plausible.The optimization includes latent transition likelihoods, stability terms, and autoregressive trajectory rollouts.
  • Optimization: Ground-plane and contact constraints reduce foot skating by encouraging likely contacting joints to remain near the shared scene ground plane.The ground plane is optimized jointly across people and timesteps.

4. Experimental Results

The experiments evaluate global trajectory reconstruction on EgoBody and PoseTrack, including ablations, state-of-the-art comparisons, qualitative results, and tracking downstream performance. Results show improved world-level reconstruction and more robust tracking, while local-pose performance depends on visibility and evaluation setting.

  • Ablation study: Removing key components, including the final motion-prior and scale optimization, significantly affects system performance, especially acceleration error.The ablation reports W-MPJPE, WA-MPJPE, and acceleration error on EgoBody.
  • Datasets and evaluation: The evaluation uses EgoBody for global 3D trajectory assessment and PoseTrack for qualitative reconstruction and downstream tracking because PoseTrack lacks 3D ground truth.EgoBody provides global-frame ground truth, whereas PoseTrack evaluation relies on tracking.
  • EgoBody results: The method comfortably outperforms GLAMR on EgoBody, achieving significant improvements in world trajectory metrics and acceleration error.GLAMR estimates world trajectories from local pose transitions without scene pixel motion, whereas this method uses relative camera motion.
  • EgoBody results: The method outperforms camera-frame baselines on world-level metrics and ranks second on local pose metrics.Compared baselines include PHALP, VIBE, and VIBE-opt.
  • PoseTrack results: On PoseTrack, the method recovers smoother, more plausible multi-person world trajectories than GLAMR and PHALP+ in qualitative comparisons.The comparisons visualize top-down trajectories across entire videos and emphasize reduced jitter and improved depth relationships.
  • PoseTrack results: Integrating recovered camera motion into PHALP improves tracking consistency under abrupt camera motion by reducing identity switches.The integration places people in the world coordinate frame using relative cameras and the optimized scale.

5. Discussion

The method recovers world-coordinate human trajectories from moving-camera videos by jointly using relative camera estimates and learned human-motion priors. It outperforms state-of-the-art methods on EgoBody and produces plausible multi-person trajectories under challenging camera motion, while remaining vulnerable to ill-posed multiview geometry.

  • Discussion: The method jointly optimizes human trajectories with relative camera estimates and learned human-motion priors in a shared world coordinate frame.This supports reconstruction from videos with moving cameras and multiple people.
  • Discussion: Predominantly rotational camera motion or co-linear human-camera motion can produce inconsistent recovered trajectories.These cases make the multiview geometry ill-posed.

A. Details of EgoBody evaluation

The EgoBody evaluation uses validation videos with dynamic cameras and ground-truth global human motion, applying common detections, keypoints, and camera intrinsics to the proposed method and baselines. It reports both local-pose and global-trajectory metrics, including Procrustes-aligned evaluations.

  • A. Details of EgoBody evaluation: The evaluation uses EgoBody validation videos, DROID-SLAM camera estimates with ground-truth intrinsics, and PHALP+ tracklets associated using the ground-truth 3D pose.The proposed method and baselines use the same detected tracklets, bounding boxes, 2D keypoints, and ground-truth intrinsics.
  • A. Details of EgoBody evaluation: Sequences are split into 100-frame segments, which are optimized separately to accelerate inference.This segmentation is part of the evaluation procedure.
  • A. Details of EgoBody evaluation: The evaluation reports local pose metrics, including PA-MPJPE, and global metrics that assess the estimated trajectory across the reconstructed sequence.The global evaluation includes alignment-based settings such as Procrustes alignment.

B. Details of PoseTrack tracking experiment

The PoseTrack tracking experiment minimally modifies PHALP so camera information affects only location-related cues. World-frame coordinates replace camera-frame coordinates for tracking, while the authors identify broader 3D-location and affinity integrations as future improvements.

  • B. Details of PoseTrack tracking experiment: The experiment modifies PHALP only through camera effects on its 2D-location and nearness cues, leaving appearance and pose cues unchanged.This isolates the contribution of camera information while retaining the main tracking algorithm.
  • B. Details of PoseTrack tracking experiment: World-frame coordinates replace camera-frame coordinates for person location, with X and Y representing normalized 2D location and Z determining nearness.The rest of the PHALP pipeline remains unchanged.
  • B. Details of PoseTrack tracking experiment: The experiment does not use the available explicit 3D location as a direct tracking cue or revise PHALP’s pose-affinity metrics.Both changes are described as potential future improvements requiring additional modifications.

C. Additional implementation details

The implementation handles people on shared or separate floors, occluded tracks, and variable track lengths through clustered floor variables, visibility masking, and synchronized incremental optimization. Runtime grows linearly with the number of tracked people, reaching about 40 minutes for roughly 100 frames and four people.

  • C. Additional implementation details: People on the same floor share one floor variable, while people on different levels are clustered and assigned separate floor variables g_k.The optimization uses foot locations to form clusters when a single floor variable cannot solve the floor constraint.
  • C. Additional implementation details: During the first optimization stages, each person is optimized independently and losses are masked for invisible frames and tracks.Only visible people are optimized during these stages.
  • C. Additional implementation details: During the final stage, all tracks are optimized in one batch so scale and ground-contact information can be shared across people.Tracks are stored relative to their first appearance and padded to the longest track length.
  • C. Additional implementation details: Each track’s rollout is synchronized to its original video interval, with padded states and out-of-interval frames removed or masked.This preserves track timing within the incremental optimization scheme.
  • C. Additional implementation details: Runtime grows linearly with the number of tracked people, requiring around 40 minutes for a sequence of about 100 frames and four people.The reported runtime characterizes the computational cost of the optimization.

D. Robustness

The method improves optimization robustness for long sequences and dynamic-camera videos, while rare failures remain on challenging PoseTrack sequences.

  • Optimization robustness: Rollout-horizon optimization improves robustness for longer sequences when the HuMoR motion prior is difficult to optimize.The strategy optimizes each person’s pose sequence over a rollout horizon and can apply to motion priors modeling transitions.
  • Optimization robustness: 99% of Egobody sequences optimize successfully with imperfect camera motion, compared with HuMoR’s 70% success rate using a static-camera assumption.HuMoR fails on 30% of sequences with identity camera, whereas the proposed approach succeeds in 99%.
  • PoseTrack robustness: Rare optimization failures persist on challenging PoseTrack sequences, mostly due to the single-floor assumption.The authors suggest clustering people across different floors as a remedy.

E. Limitations

The approach remains sensitive to upstream estimation errors, tracking identity mistakes, and monocular motions that are intrinsically difficult to decouple.

  • Upstream dependencies: Failures in estimated camera motion or person tracking can propagate into the optimization.The method relies on approximate camera estimates and outputs from a person-tracking system.
  • Camera estimation: SfM can confuse translation with rotation, although the optimization typically recovers reasonable motion from imperfect camera estimates.Jointly optimizing camera and human motion would require updating the 3D structure.
  • Tracking identity: Identity-switch errors, especially merging two people into one tracklet, remain a harmful failure mode because identity is not explicitly optimized.Future work could jointly solve tracklet association and human-camera motion optimization.
  • Monocular ambiguity: Co-linear motion between people and the camera can cause underestimated location changes, making people appear to run in the same location.Background-scale priors such as monocular depth cues may better constrain these cases.
Loading 2302.12827v2…