Source-linked AI summary
World-Grounded Human Motion Recovery via Gravity-View Coordinates
Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng, Zechen Hu, Hujun Bao, Ruizhen Hu, Xiaowei Zhou
TL;DR
World-grounded human motion recovery must handle ambiguous world-coordinate definitions and accumulated errors in autoregressive approaches. GVHMR predicts poses in a gravity- and view-defined coordinate system, then recovers global motion through camera rotations. Across benchmarks, it reports state-of-the-art accuracy and motion plausibility in both camera-space and world-grounded metrics.
Problem
World-grounded motion recovery is difficult because valid gravity-aware world coordinates can vary by rotation around gravity, while existing approaches accumulate errors or remain camera-space.
Method
GVHMR predicts gravity-aware human poses per frame in a Gravity-View coordinate system and transforms them into a consistent global trajectory using relative camera rotations.
Results
GVHMR achieves state-of-the-art accuracy and motion plausibility across camera-space and world-grounded metrics, outperforming existing methods.
Takeaways & Limitations
Per-frame GV estimation and gravity-constrained composition provide a route to arbitrary-length world-grounded motion recovery without autoregressive error accumulation.
Takeaways & Limitations
Camera-space methods remain limited because their predictions can become physically implausible when the camera moves.
Abstract
from arXiv · showhide
We present a novel method for recovering world-grounded human motion from monocular video. The main challenge lies in the ambiguity of defining the world coordinate system, which varies between sequences. Previous approaches attempt to alleviate this issue by predicting relative motion in an autoregressive manner, but are prone to accumulating errors. Instead, we propose estimating human poses in a novel Gravity-View (GV) coordinate system, which is defined by the world gravity and the camera view direction. The proposed GV system is naturally gravity-aligned and uniquely defined for each video frame, largely reducing the ambiguity of learning image-pose mapping. The estimated poses can be transformed back to the world coordinate system using camera rotations, forming a global motion sequence. Additionally, the per-frame estimation avoids error accumulation in the autoregressive methods. Experiments on in-the-wild benchmarks demonstrate that our method recovers more realistic motion in both the camera space and world-grounded settings, outperforming state-of-the-art methods in both accuracy and speed. The code is available at https://zju3dv.github.io/gvhmr/.
1 INTRODUCTION
World-grounded human motion recovery seeks gravity-aware 3D motion, but ambiguous world coordinates and accumulated errors make consistent recovery difficult. GVHMR addresses this with Gravity-View coordinates, per-frame estimation, and sequence modeling designed for long videos.
- World-grounded motion is intended as gravity-aware foundational data for text-to-motion generation and humanoid robot imitation learning.
- Camera-to-world transformations can accumulate translation and pose errors, while autoregressive recovery requires good initialization and degrades over long sequences.
- GVHMR defines a Gravity-View coordinate system from gravity and camera-view directions, predicts gravity-aware poses per frame, and recovers relative rotations between frames.
- A RoPE-enhanced relative transformer directly regresses entire motion sequences, while receptive-field masking enables parallel inference without sliding windows.
- GVHMR combines GV coordinates, global orientation recovery, RoPE-based sequence modeling, and extensive evaluations showing stronger camera-space and world-grounded accuracy than prior methods.
2 RELATED WORKS
Prior human-motion methods primarily recover camera-space poses, whereas world-grounded recovery requires gravity-aware alignment and additional calibration, sensing, or camera-motion estimation. GVHMR instead processes monocular-video cues through a per-frame pipeline and transforms its intermediate representations into global motion.
- Camera-space HMR methods estimate poses from images or videos but can produce physically implausible motion when the camera moves.
- Traditional world-grounded recovery uses floor-plane calibration, multi-camera systems, or IMU sensors to estimate gravity-aware motion and scale.
- WHAM autoregressively predicts per-frame global pose and translation but depends on good initialization and suffers error accumulation during long-term recovery.
- GVHMR preprocesses monocular video with bounding-box tracking, 2D keypoints, image features, and relative camera rotation before predicting per-frame representations and camera-frame SMPL parameters.
- Other world-grounded approaches transform camera-coordinate results using visual odometry or SLAM, sometimes adding trajectory refinement or scene-based scale estimation.
3 METHOD
GVHMR represents each frame in a gravity-aligned Gravity-View coordinate system, then transforms per-frame predictions into a consistent world trajectory. A RoPE-enhanced Relative Transformer processes fused video features and supports parallel inference over long sequences.
- Global trajectory representation: The task predicts local SMPL-X body poses, shape coefficients, camera-space trajectory, and world-space trajectory from video.The predicted outputs include local poses, shape, camera orientation and translation, and world orientation and translation.
- Global trajectory representation: Gravity-View coordinates use gravity as the y-axis and construct the remaining axes from the camera view direction, resolving rotational ambiguity for per-frame orientation prediction.The x-axis is perpendicular to gravity and the view direction, while the z-axis follows the right-hand rule.
- Global trajectory representation: Each frame has an independent GV system, and GV0 serves as the world reference while adjacent systems are aligned using relative rotations around the gravity axis.This formulation converts per-frame orientations into a common gravity-aware global trajectory and limits the relative rotation to one degree of freedom.
- Global trajectory representation: The method predicts local root velocities and transforms them using recovered orientations before cumulatively composing global translation.For moving cameras, relative camera rotations are used to recover the relative rotation between adjacent GV systems.
- Global trajectory representation: The GV formulation avoids cumulative gravity-direction errors and allows parallel orientation processing, while the method does not require initialization or autoregressive prediction.The paper attributes this to consistency of the GV y-axis and per-frame estimation.
- Network design: The network fuses bounding boxes, 2D keypoints, image features, and relative camera rotations into per-frame tokens processed by a RoPE-enhanced Relative Transformer.Multitask MLPs produce intermediate representations, and the resulting GV predictions and root velocities are converted to world-frame motion.
- Network design: A receptive-field-limited attention mask restricts each token to nearby temporal positions, enabling generalization to arbitrarily long sequences without autoregressive sliding-window inference.RoPE injects relative temporal features, while the mask uses the maximum training length as the receptive-field limit.
4 EXPERIMENTS
Experiments evaluate GVHMR on in-the-wild benchmarks for world-grounded and camera-space motion recovery, ablations, test-time augmentation, and runtime. The method achieves strong accuracy and long-sequence robustness while substantially reducing core inference time relative to WHAM and optimization-based methods.
- Datasets and Metrics: GVHMR is evaluated on 3DPW, RICH, and EMDB, using separate splits for world-grounded and camera-space performance.RICH and EMDB-2 assess global motion, while RICH, EMDB-1, and 3DPW assess camera-coordinate motion.
- Global Motion Recovery: GVHMR achieves the best performance on all reported world-grounded metrics and better handles relative camera-rotation errors than WHAM.On EMDB with dynamic cameras, replacing gyro input with DPVO causes only a 1.6mm/0.1% drop in W-MPJPE100/RTE, compared with WHAM’s 19.5mm/1.9% drop.
- Global Motion Recovery: GVHMR recovers more plausible global motion than WHAM, maintaining much lower global-orientation error during long-term prediction.The comparison specifically highlights improved stability as sequence length increases.
- Camera-Space Motion Recovery: In camera-space recovery, GVHMR achieves the best performance on most metrics and remains competitive with WHAM on 3DPW.Its PA-MPJPE is 0.3 mm behind WHAM on 3DPW, while the authors attribute the difference to predicting SMPLX rather than SMPL parameters.
- Ablation Studies: Removing GV coordinates slightly degrades camera-space metrics, while replacing RoPE with absolute positional encoding produces very poor long-sequence results.A sliding-window variant with absolute positional encoding remains worse than the proposed approach.
- Dataset and Test-Time-Augmentation Ablation: On BEDLAM-trained EMDB experiments, GVHMR significantly outperforms WHAM, while FlipEval improves camera-space metrics more than global-space metrics.This comparison holds with or without FlipEval test-time augmentation.
- Running Time: For a 1430-frame video, GVHMR’s core network takes 0.28 seconds, compared with 2.0 seconds for WHAM and more than 6 hours for SLAHMR.These timings exclude preprocessing and were measured on an RTX 4090 GPU.
5 CONCLUSIONS
GVHMR recovers world-grounded human motion from monocular videos using Gravity-View coordinates and a RoPE-enhanced relative transformer. Experiments show state-of-the-art accuracy and motion plausibility in both camera-space and world-grounded metrics, while supporting arbitrary-length inference without sliding windows.
- Contributions: GVHMR defines Gravity-View coordinates to use gravity priors and constraints, avoiding error accumulation along the gravity axis.The coordinate system is paired with global orientation recovery for world-grounded motion.
- Contributions: A relative transformer with RoPE enables inference on sequences of arbitrary length without sliding windows.This design supports the paper’s long-sequence motion-recovery setting.
- Results: GVHMR achieves state-of-the-art accuracy and motion plausibility across benchmarks in both camera-space and world-grounded metrics.The conclusion reports this outcome across the paper’s evaluated settings.