Source-linked AI summary
Learning the Depths of Moving People by Watching Frozen People
Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, William T. Freeman
TL;DR
Dense depth prediction for monocular videos with moving cameras and people is difficult because dynamic objects violate common geometric assumptions and existing methods may be sparse. The paper learns human depth priors from MannequinChallenge videos of stationary people, guides prediction with motion-parallax depth, and reports improved accuracy over state-of-the-art methods while enabling 3D effects.
Problem
Dense depth for monocular videos with freely moving cameras and people remains difficult because existing dynamic-object methods impose strong motion assumptions or recover sparse depth.
Method
The method learns human depth priors from videos of people frozen in natural poses, uses SfM and MVS for supervision, and feeds motion-parallax environment depth into a dense-depth network.
Results
The model predicts depth more accurately than state-of-the-art monocular depth prediction and motion stereo methods on real-world videos of complex human actions.
Takeaways & Limitations
The predicted depth supports synthetic depth-of-field, depth-aware inpainting, and virtual-object insertion with correct occlusion.
Takeaways & Limitations
The approach assumes known camera poses, may be inaccurate for non-human moving regions, and can produce temporally inconsistent estimates because it uses only two views.
Abstract
from arXiv · showhide
We present a method for predicting dense depth in scenarios where both a monocular camera and people in the scene are freely moving. Existing methods for recovering depth for dynamic, non-rigid objects from monocular video impose strong assumptions on the objects' motion and may only recover sparse depth. In this paper, we take a data-driven approach and learn human depth priors from a new source of data: thousands of Internet videos of people imitating mannequins, i.e., freezing in diverse, natural poses, while a hand-held camera tours the scene. Because people are stationary, training data can be generated using multi-view stereo reconstruction. At inference time, our method uses motion parallax cues from the static areas of the scenes to guide the depth prediction. We demonstrate our method on real-world sequences of complex human actions captured by a moving hand-held camera, show improvement over state-of-the-art monocular depth prediction methods, and show various 3D effects produced using our predicted depth.
1. Introduction
The paper targets dense depth prediction for videos where both the camera and people move, using learned human depth priors from stationary-person videos and motion-parallax cues.
- Dynamic people violate the epipolar constraints used in 3D vision, making dense geometry recovery from hand-held video challenging.
- The method learns human shape and deformation priors from data rather than explicitly imposing assumptions on articulated human motion.Humans are emphasized because they are salient in applications such as augmented reality and difficult to model.
- Thousands of YouTube videos show people frozen in natural poses while a hand-held camera moves, enabling stationary-scene reconstruction for training.The resulting MannequinChallenge dataset addresses the difficulty of obtaining diverse, naturally moving human training data at scale.
- The network takes an RGB image, human-region mask, and initial non-human depth from two-frame motion parallax, then predicts dense depth for people and environment.The motion-parallax input supplies information unavailable from a single frame, and the trained model handles arbitrary camera and human motion.
- The model achieves higher depth accuracy than state-of-the-art monocular depth prediction and motion stereo methods on real-world videos of complex human actions.Predicted depths also support synthetic depth-of-field, depth-aware inpainting, and virtual-object insertion with correct occlusion.
- The paper contributes a new frozen-person video source for depth prediction and a deep model for simultaneous camera motion and complex human motion.
2. Related Work
Prior work estimates depth from images, RGBD data, or motion decomposition, but does not adequately provide dense depth for naturally moving people in monocular video.
- Learning-based depth prediction: Learning-based depth methods predict dense depth from single or multiple RGB images, sometimes assuming known camera poses or jointly estimating poses and depth.
- Learning-based depth prediction: Existing learning-based depth methods are not designed to predict the depth of dynamic objects.
- Depth estimation for dynamic scenes: Monocular dynamic-scene methods often reconstruct only sparse geometry or decompose scenes into piecewise rigid parts using motion or object segmentation.
- RGBD data for learning depth: RGBD datasets generally lack depth supervision for moving people in natural environments, while action datasets mostly use static cameras and limited indoor scenes.
- Human shape and pose prediction: Single-image human-mesh methods model the body but disregard hair, clothing, and non-human scene regions, and often require most of the body to be visible.
3. MannequinChallenge Dataset
The MannequinChallenge dataset uses videos of stationary people and moving cameras to reconstruct training depth, then filters noisy reconstructions and unsuitable clips.
- The dataset derives static-scene geometry from YouTube videos where people freeze while a camera operator moves around them.Stationarity permits camera-pose and depth estimation with structure-from-motion and multi-view stereo.
- The videos cover diverse scenes, ages, poses, and group configurations, with around 2,000 candidate videos identified as processable.
- Camera poses are initialized with ORB-SLAM2 and refined at higher resolution using a visual structure-from-motion system.
- Dense per-frame depth maps are reconstructed using COLMAP, a multi-view stereo system.
- Raw multi-view stereo depths are filtered because Internet videos contain motion blur, shadows, reflections, and other sources of noise.Filtering combines depth refinement with consistency against two-frame motion-parallax depth.
- Depth values with normalized discrepancy ∆(p) > 0.2 between MVS and motion-parallax estimates are removed.The discrepancy compares DMVS with Dpp for each valid pixel.
- Frames and clips are removed when people move, backgrounds are synthetic, valid MVS coverage is low, camera parameters are unreliable, or reconstructions are obviously incorrect.Additional requirements include sequences of at least 30 frames, 16:9 aspect ratio, and width of at least 1600 pixels.
- The processed dataset contains 4,690 sequences and more than 170K valid image-depth pairs, split 80:3:17 across training, validation, and testing clips.
4. Depth Prediction Model
The model combines RGB, human masks, motion-parallax depth, confidence, and optional keypoints to predict dense depth for entire scenes. It is trained against filtered MVS depth with losses designed for scale invariance, sharp discontinuities, and smooth interpolation.
- Model inputs: The network predicts full-scene depth from an RGB image, human-region mask, masked environmental depth, confidence map, and optional keypoints.During training and inference, camera poses are assumed known; the model inpaints human depth and refines non-human depth.
- Depth from motion parallax: Motion parallax between reference and source frames provides the initial depth estimate for static, non-human regions.Optical flow and relative camera poses are combined using the Plane-Plus-Parallax representation.
- Keyframe selection: Reference-source pairs are selected to balance significant view overlap with sufficient camera baseline.Pairs with co-visible-feature fraction below τo = 0.6 are discarded, and the maximum frame interval is limited to 10.
- Confidence: The confidence map reflects flow consistency, epipolar compliance, and parallax, helping the network weight reliable versus unreliable input depth.Human regions and pixels with confidence C(p) < 0.25 are masked out; epipolar tolerance is set to 2px and parallax tolerance to 1°.
- Results: Additional environmental depth and confidence inputs improve performance in human and non-human regions, while optical flow alone is only on par with the single-view model.Qualitative comparisons also report improved depth relations between people and their surroundings for the full model.
- Losses: The scale-invariant loss combines log-space depth error, multi-scale gradients, and edge-aware smoothness.The gradient term supports sharp depth discontinuities and smooth changes, while smoothness regularizes textureless regions where MVS fails.
5. Results
The method improves dense depth prediction for moving-camera scenes with people, outperforming competing approaches quantitatively and qualitatively while supporting depth-based visual effects.
- The evaluation compares the method with single-view, motion-based, and dynamic-scene depth prediction algorithms on TUM RGBD and Internet videos.The experiments include 11 TUM RGBD sequences totaling 1,815 images and challenging Internet clips with natural camera and human motion.
- Adding environment depth and confidence information improves accuracy in both human and non-human regions, while human keypoints provide further gains.Optical-flow input is only on par with the single-view model because the network cannot implicitly recover relative camera poses from flow alone.
- The full model significantly improves performance across all reported error measures by using the initial masked environment depth.Replacing this input with optical flow yields performance only slightly better than the single-view network.
- On TUM RGBD, the predicted depth strongly resembles sensor ground truth, with detailed structure and sharp depth discontinuities.The method improves errors in both human regions, such as legs, and non-human regions, such as tables and ceilings.
- On Internet videos, the method produces more accurate depth ordering between people and surroundings and within human regions than the baseline methods.The baselines struggle with Internet-video generalization, human regions, or their static-scene assumptions.
- The predicted depth maps enable depth-aware defocus, insertion of synthetic 3D graphics, and removal of nearby people through inpainting.These effects demonstrate uses of the estimated depth beyond numerical evaluation.
6. Discussion and Conclusion
The paper demonstrates a learning-based approach for dense depth prediction in dynamic scenes and introduces Mannequin Challenge videos as training data. It also identifies limitations involving camera poses, moving non-human regions, and temporal consistency.
- The method predicts dense depth for scenes where a monocular camera and people move freely.
- Mannequin Challenge videos provide a large training corpus of people frozen in natural poses while the camera moves.The authors derive reliable depth supervision from this noisy data.
- Known camera poses may be difficult to infer when moving objects cover most of the scene.
- Depth may be inaccurate for moving non-human regions such as cars and shadows, and two-view processing can produce temporal inconsistencies.