Source-linked AI summary
Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera
Weipeng Xu, Avishek Chatterjee, Michael Zollhoefer, Helge Rhodin, Pascal Fua, Hans-Peter Seidel, Christian Theobalt
TL;DR
Mobile 3D pose estimation in unconstrained activities requires a portable, real-time, and robust capture system. The paper uses a lightweight baseball-cap-mounted fisheye camera with a tailored CNN-based method and training corpus, reporting improved pose accuracy and overlay over baselines. Its main scope boundaries are limited generalization beyond the training distribution, ambiguity under strong occlusion, and possible temporal instability.
Problem
Mobile 3D pose estimation for unconstrained activities must support mobility, real-time operation, long sequences, and recovery from tracking failures.
Method
The paper combines a lightweight single cap-mounted fisheye camera with a CNN-based disentangled estimator and synthetic top-down fisheye training corpus.
Results
The authors report lower 3D joint error and better 2D overlay than existing baselines in the egocentric fisheye setting.
Takeaways & Limitations
The setup is presented as an important step toward practical daily full-body motion capture and applications including action recognition, performance analysis, and motion control.
Takeaways & Limitations
The method generalizes poorly outside its training-corpus distribution, struggles with strongly occluded poses, and may produce temporally unstable framewise predictions.
Abstract
from arXiv · showhide
We propose the first real-time approach for the egocentric estimation of 3D human body pose in a wide range of unconstrained everyday activities. This setting has a unique set of challenges, such as mobility of the hardware setup, and robustness to long capture sessions with fast recovery from tracking failures. We tackle these challenges based on a novel lightweight setup that converts a standard baseball cap to a device for high-quality pose estimation based on a single cap-mounted fisheye camera. From the captured egocentric live stream, our CNN based 3D pose estimation approach runs at 60Hz on a consumer-level GPU. In addition to the novel hardware setup, our other main contributions are: 1) a large ground truth training corpus of top-down fisheye images and 2) a novel disentangled 3D pose estimation approach that takes the unique properties of the egocentric viewpoint into account. As shown by our evaluation, we achieve lower 3D joint error as well as better 2D overlay than the existing baselines.
1 INTRODUCTION
The paper targets real-time mobile 3D pose estimation in unconstrained activities, where capture systems must be mobile, fast, robust over long sequences, and able to recover from failures. It introduces a lightweight cap-mounted fisheye setup and tailored estimation approach addressing the egocentric viewpoint.
- Unconstrained activities require mobile capture, real-time performance, long-sequence robustness, and rapid recovery from tracking failures.
- Static outside-in systems are impractical for daily activities because they are immobile, location-dependent, occlusion-sensitive, and limited in recording volume.
- Existing mobile alternatives can be expensive, calibration-intensive, computationally slow, uncomfortable, or insufficiently robust for long full-body sequences.
- The proposed setup converts a standard baseball cap into a compact, lightweight, power-efficient device using one fisheye camera, without actor calibration and with dynamic-background support.
- 60 Hz pose estimation and online frame-by-frame processing support long captures while enabling automatic recovery from occasional failures.
- Training data and conventional estimators do not match the top-down, radially distorted viewpoint, while the proposed disentangled method preserves 2D-3D consistency.
- The authors report the first real-time mobile 3D pose system using a single egocentric fisheye camera and better performance than baseline methods on their test set.
2 RELATED WORK
Prior motion-capture systems trade off accuracy, mobility, cost, calibration, recording volume, or intrusiveness. The paper positions its single-camera cap setup as a lightweight alternative for full-body capture in everyday settings.
- Studio and Multi-view Motion Capture: Studio and multi-view systems can achieve high accuracy but require costly, time-consuming setups and impose constrained recording volumes.
- Studio and Multi-view Motion Capture: Static-camera systems can fail under occlusion, while hand-held mobile systems require additional operators.
- Monocular Human Pose Estimation: Monocular pose estimation supports consumer applications, but existing RGB-D and RGB approaches differ in sensing assumptions, processing strategies, and camera models.
- Body-worn Motion Sensors: Body-worn sensors and exoskeletons enable mobile capture but require subject instrumentation and calibration, increasing setup time and complicating capture across many people.
- Mobile Motion Capture: Inside-out camera systems support free roaming, yet body-mounted arrangements make accurate full-body observation difficult from close proximity.
- Mobile Motion Capture: Stereo fisheye systems enable free roaming but use bulky hardware and provide interactive rather than full-body real-time tracking.
3 THE MO2CAP2 APPROACH
Mo2Cap2 combines a lightweight cap-mounted fisheye camera with a synthetic training corpus and a disentangled estimator tailored to egocentric distortion. Its pipeline separates 2D detection, distance estimation, and calibrated back-projection to produce consistent 3D joint positions.
- Lightweight Hardware Setup: A single 175g fisheye camera mounted 8cm from a baseball cap provides a lightweight, comfortable setup with a 182° field of view.The wide view captures the full body across varied motions, while the close, oblique monocular viewpoint makes 3D estimation challenging.
- Monocular Fisheye 3D Pose Estimation: The estimator uses two 2D branches, a per-joint distance regressor, and a joint-position module that back-projects detections using fisheye calibration.The branches process the full image and a lower-body zoom-in to address the lower body’s reduced image scale.
- Monocular Fisheye 3D Pose Estimation: The joint-position module multiplies each calibrated ray direction by its predicted absolute joint-to-camera distance to recover 3D positions.This design preserves exact reprojection to the detected 2D joint locations and uses location-dependent information as a depth cue.
- Monocular Fisheye 3D Pose Estimation: The method is designed to improve 3D accuracy by handling upper–lower-body scale differences and enforcing 2D–3D consistency.The implementation uses residual networks and intermediate supervision for heatmap prediction.
4 RESULTS
Mo2Cap2 is evaluated on unconstrained real-world activities and annotated indoor/outdoor sequences, with comparisons against existing architectures and an ablation of its zoom-in branch. The system runs in real time, improves reconstruction accuracy and image overlay, but remains limited by distribution shift and occlusion-related ambiguity.
- Runtime: 60 Hz on an Nvidia GTX 1080 Ti enables real-time use where motion-control latency matters.The forward pass takes 16.7 ms.
- Qualitative results: Five unconstrained activities show accurate 3D pose estimates, including interactions with people or objects.The activities are making tea, office work, football, bicycling, and juggling, with approximately 2000 frames per sequence.
- Quantitative results: The quantitative test set contains 8 actions across 5591 indoor and outdoor frames with commercial multi-view motion-capture ground truth.Actions include walking, sitting, crawling, crouching, boxing, dancing, stretching, and waving.
- Quantitative results: The disentangled approach outperforms vectorized-pose and location-map baselines in mean joint error on real-world sequences.The baselines are trained on the authors’ dataset and adapted for a fair architectural comparison.
- Overlay comparison: The proposed results accurately overlay the input images, whereas baseline methods exhibit significant offsets.The comparison concerns 3D pose results overlaid on the images.
- Ablation and limitations: The zoom-in branch significantly improves lower-body-joint overlay, while strong occlusions can produce incorrect poses.Reported failures include predicting standing instead of sitting when the legs are fully occluded and aligning a barely visible arm to a cupboard edge.
- Ablation and limitations: The method does not generalize well to data far outside its training corpus, although expanding motion, body-shape, and appearance variation is proposed as mitigation.Strong occlusions remain ambiguous because multiple body poses can produce the same observation.
5 APPLICATIONS
The egocentric pose system supports action recognition, motion control, and performance analysis, with particular relevance to controller-free VR and AR interaction. Its inside-in design can integrate with headsets without external tracking devices.
- Applications: The system supports action recognition, motion control, and performance analysis across sports, health-care, and related applications.The paper specifically presents body-gesture interaction as a use case for VR and AR.
- Applications: In VR and AR, body-gesture interaction can provide compact, controller-free full-body control without external tracking devices.The system is described as an inside-in solution that can be integrated into VR headsets.
6 CONCLUSION
The paper concludes that a cap-mounted single-fisheye system, paired with a tailored regression network and synthetic top-down training corpus, advances practical mobile full-body capture. Its full method improves lower-body overlay and outperforms existing baselines on the egocentric setting.
- Conclusion: A single fisheye camera attached to a standard baseball cap provides a real-time approach toward practical daily full-body motion capture.The authors characterize the setup as a novel monocular alternative to more cumbersome technologies.
- Conclusion: A zoom-in branch yields significantly better overlay of lower-body joints in the full method.The improvement is attributed to the zoom-in branch in the 2D module.
- Conclusion: Under strong occlusion, the method can output incorrect poses, including standing instead of sitting or aligning an arm to nearby scene structure.These are presented as typical failure cases.
- Conclusion: The tailored regression network is trained on synthetic top-down fisheye images and achieves lower 3D joint error and better 2D overlay than existing baselines.The paper positions these results specifically within the egocentric fisheye setting.