Source-linked AI summary
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors
Vladimir Guzov, Aymen Mir, Torsten Sattler, Gerard Pons-Moll
TL;DR
HPS addresses the challenge of recovering full-body 3D human pose while registering people to large scanned environments using only wearable sensors. It fuses IMU tracking, camera self-localization, and scene constraints in a joint optimization, producing drift-free, natural motion tracking across large scenes. The accompanying dataset supports egocentric tracking and human–scene interaction research, though prolonged camera-localization errors can degrade performance.
Problem
Existing third-person capture is constrained by occlusions and limited recording areas, while HPS targets full-body pose and scene registration using wearable sensors.
Method
HPS jointly optimizes IMU-based body pose, head-camera self-localization, and scene constraints using a pre-scanned 3D environment.
Results
HPS produces natural, drift-free human motion tracking continuously across large scenes, including multiple rooms and outdoors.
Takeaways & Limitations
The HPS dataset provides egocentric video, IMU data, 3D scans, and reconstructed humans for studying tracking and interaction in large scenes.
Takeaways & Limitations
HPS performance deteriorates when camera localization remains inaccurate for long periods, with reported errors ranging from 4cm to 15cm.
Abstract
from arXiv · showhide
We introduce (HPS) Human POSEitioning System, a method to recover the full 3D pose of a human registered with a 3D scan of the surrounding environment using wearable sensors. Using IMUs attached at the body limbs and a head mounted camera looking outwards, HPS fuses camera based self-localization with IMU-based human body tracking. The former provides drift-free but noisy position and orientation estimates while the latter is accurate in the short-term but subject to drift over longer periods of time. We show that our optimization-based integration exploits the benefits of the two, resulting in pose accuracy free of drift. Furthermore, we integrate 3D scene constraints into our optimization, such as foot contact with the ground, resulting in physically plausible motion. HPS complements more common third-person-based 3D pose estimation methods. It allows capturing larger recording volumes and longer periods of motion, and could be used for VR/AR applications where humans interact with the scene without requiring direct line of sight with an external camera, or to train agents that navigate and interact with the environment based on first-person visual input, like real humans. With HPS, we recorded a dataset of humans interacting with large 3D scenes (300-1000 sq.m) consisting of 7 subjects and more than 3 hours of diverse motion. The dataset, code and video will be available on the project page: http://virtualhumans.mpi-inf.mpg.de/hps/ .
1. Introduction
HPS addresses the limitations of third-person capture by combining wearable sensing with scene registration to recover full-body 3D motion over large areas and long periods. It also provides an egocentric dataset for studying human motion and interaction in large environments.
- HPS enables egocentric capture without external cameras, supporting interaction with virtual objects and motion beyond restricted recording volumes.
- Third-person capture is limited by occlusions, restricted viewing areas, and typically short recordings.
- HPS recovers full-body 3D pose registered to a large 3D scene using body-mounted IMUs and an outward-facing head camera.
- Jointly integrating IMU tracking, camera self-localization, and scene constraints removes drift and supports natural motion over large areas and long periods.
- The HPS dataset includes 8 environment types, 7 subjects, and varied activities in scenes, including some larger than 1000m2.
2. Related Work
Related work spans IMU and egocentric pose capture, camera localization, and human–scene reasoning. HPS differs by combining outward-facing egocentric sensing with scene registration for full-body motion across multiple rooms and outdoor environments.
- IMU-based 3D Human Pose Estimation: IMU-based pose systems reduce sensor requirements but still suffer from drift, especially in global body orientation and location.
- Ego-centric capture and prediction: Most egocentric capture methods view the body from a head-mounted camera, while outward-facing methods estimate 3D pose with high uncertainty.
- Ego-centric capture and prediction: Multi-camera structure-from-motion can recover outward-facing egocentric pose but only for slow motions.
- Camera Localization: Structure-based localization is precise in small scenes but does not scale well, whereas image-based methods are more scalable but less precise.
- Camera Localization: Hybrid localization combines image retrieval with structure-based pose recovery, providing precision and scalability for large scenes.
- Humans and Scenes: Unlike approaches that mostly analyze static poses from third-person views, HPS captures moving humans from a first-person perspective across multiple rooms and outdoor environments.
3. Method
HPS combines head-camera self-localization, body-mounted IMU tracking, and a 3D scene model to estimate a human’s full-body pose and position in a known environment. A joint optimization balances noisy but drift-free camera estimates with short-term accurate IMU articulation while enforcing scene contact and motion smoothness.
- Overview: The method estimates full-body pose and location from a head-mounted camera, body-mounted IMUs, and a pre-built 3D scene with calibrated RGB images.The pipeline localizes the person, estimates pose from IMUs, and jointly optimizes these estimates with scene constraints.
- Camera self-localization: Camera self-localization retrieves relevant database images, forms 2D-3D feature matches, and estimates camera pose by minimizing reprojection error.NetVLAD retrieves images, while SuperPoint features and rendered scene geometry provide the correspondences.
- IMU tracking: IMU tracking provides accurate body articulation but suffers from significant global orientation and position drift over time.The system uses IMU pose estimates mainly for body articulation rather than unreliable global orientation and translation.
- Joint optimization: The joint optimization enforces camera self-localization, scene, and smoothness constraints while remaining close to the IMU body-pose estimate.The optimization runs over a batch of frames and excludes global orientation and position from the IMU closeness term.
- Scene constraints: When foot contact is detected, scene terms constrain foot vertices to nearby scene points and prevent sliding across successive frames.The contact term uses toe and heel vertex sets and includes contact and sliding subterms.
- Initialization: Initialization removes camera-localization outliers using a 3 m/s velocity threshold, interpolates replacements, and aligns IMU pose orientation with the camera trajectory.The orientation alignment uses estimated trajectory tangents; stationary frames reuse the last correction with non-zero velocity.
4. Dataset
The HPS dataset captures people performing diverse activities across large 3D scenes from an egocentric viewpoint, with synchronized visual and reference-pose data.
- 7 people perform diverse activities across 8 large scenes, including environments larger than 1000 m2.
- The dataset includes head-mounted-camera images coupled with reference 3D pose and location in pre-scanned scenes.
- More than 300K synchronized RGB images are paired with reference 3D pose and location.
- Activities include exercising, reading, eating, lecturing, computer use, making coffee, and dancing.
5. Experiments
Experiments evaluate HPS against component baselines using dynamic depth-camera ground truth and tests of drift, camera outliers, and foot-scene contact. HPS maintains accurate, drift-free motion and physically plausible contacts, while persistent localization failures require a modified method.
- Drift and Outliers: HPS does not drift with time or distance, remains robust to non-persistent camera-localization outliers, and satisfies foot-scene constraints.
- Quantitative Evaluation: Dynamic accuracy is measured by comparing the translated SMPL mesh with ground-truth point clouds from synchronized external depth cameras.
- Quantitative Evaluation: HPS is evaluated against pure IMU, IMU-plus-camera, filtered IMU-plus-camera, and no-scene-constraint baselines.
- Limitations: Persistent camera-localization failures in some outdoor and repetitive-pattern indoor scenes require a modified HPS version.
- Foot Contacts: HPS preserves foot contact better than the baselines and produces slightly less foot sliding than the raw IMU tracker.
- Qualitative Evaluation: Foot-contact constraints prevent the human mesh from flying above or penetrating the ground, producing more stable and physically correct motion.
6. Conclusions and Future Work
HPS enables continuous, drift-free human tracking in large, pre-scanned environments, while identifying localization conditions that limit accuracy. The authors position the system and released dataset as a foundation for future egocentric scene-and-human modeling.
- HPS continuously tracks humans across large scenes spanning 300–1000m2, including multiple rooms and outdoor areas.
- HPS error does not accumulate with time or distance traveled, but performance deteriorates when camera localization remains inaccurate for long periods.Reported errors range from 4cm to 15cm; localization is affected by missing features and changes between the static scan and real images.
- Applications requiring errors below 1cm remain beyond the demonstrated accuracy of HPS.
- Future work: Future improvements include dynamic local maps, semantic camera localization, additional body-part scene constraints, and models that anticipate human intent.
- Future work: The authors will release the HPS dataset and code to support research on perceiving and modeling scenes and humans from an egocentric perspective.