Source-linked AI summary
Direct Prediction of 3D Body Poses from Motion Compensated Sequences
Bugra Tekin, Artem Rozantsev, Vincent Lepetit, Pascal Fua
TL;DR
Monocular 3D pose recovery remains difficult because projection ambiguities such as self-occlusion and mirroring confuse single-frame estimates. The paper directly regresses the central-frame pose from consecutive-frame volumes, compensating motion to keep the subject centered. It improves state-of-the-art performance across Human3.6m, HumanEva, and KTH Multiview Football benchmarks.
Problem
Monocular video-based 3D pose recovery is challenging because projection ambiguities can confuse estimates, while post-hoc temporal consistency is insufficient for errors spanning several frames.
Method
The method directly regresses the central-frame 3D pose from consecutive-frame bounding-box volumes and uses CNNs to motion-compensate them so the subject remains centered.
Results
The framework improves upon state-of-the-art methods by a large margin on Human3.6m, HumanEva, and KTH Multiview Football benchmarks.
Takeaways & Limitations
Early use of motion and appearance cues from rectified spatiotemporal volumes helps disambiguate poses involving mirroring and self-occlusion.
Takeaways & Limitations
Direct 3D CNNs over pixel-intensity volumes showed no notable improvement over spatial CNNs in the authors’ experiments, likely because of model complexity and input dimensionality.
Abstract
from arXiv · showhide
We propose an efficient approach to exploiting motion information from consecutive frames of a video sequence to recover the 3D pose of people. Previous approaches typically compute candidate poses in individual frames and then link them in a post-processing step to resolve ambiguities. By contrast, we directly regress from a spatio-temporal volume of bounding boxes to a 3D pose in the central frame. We further show that, for this approach to achieve its full potential, it is essential to compensate for the motion in consecutive frames so that the subject remains centered. This then allows us to effectively overcome ambiguities and improve upon the state-of-the-art by a large margin on the Human3.6m, HumanEva, and KTH Multiview Football 3D human pose estimation benchmarks.
1. Introduction
Monocular 3D pose recovery remains difficult because 2D projections create ambiguities that post-processing cannot always resolve. The paper instead regresses pose from motion-aware, motion-compensated spatio-temporal volumes.
- Monocular 3D body pose recovery from ordinary video remains challenging despite its potential applications in athletic training, surveillance, and entertainment.
- Self-occlusion and mirroring create projection ambiguities that linking candidate poses afterward can resolve only to a limited extent.
- The proposed regressor predicts the central-frame 3D pose directly from bounding boxes in consecutive frames before and after that frame.
- The best results come from applying a Deep Network to spatiotemporal features extracted from the image volume.
- Motion compensation aligns successive bounding boxes so the person remains centered, using CNNs to predict and refine body shifts.
- The framework combines appearance and motion cues and improves upon state-of-the-art methods by a large margin on three 3D pose benchmarks.
2. Related Work
Prior 3D pose methods use still images or image sequences, but single-image approaches remain ambiguous and sequence trackers can fail. This work uses motion-compensated consecutive frames without requiring multiple cameras.
- 3D human pose estimation methods are organized into still-image and image-sequence approaches.
- 3D Human Pose Estimation in Single Images: Generative single-image methods search for plausible skeleton configurations but remain competitive only with sufficiently good initialization.
- 3D Human Pose Estimation in Single Images: Discriminative single-image regressors map image evidence directly to 3D poses but still suffer from self-occlusion, mirroring, and foreshortening.
- 3D Human Pose Estimation in Image Sequences: Unlike multi-view feature fusion, this method uses no multiple cameras and compensates for apparent body motion before collecting consecutive-frame cues.
- 3D Human Pose Estimation in Image Sequences: Frame-to-frame trackers rely on previous-frame dependencies, require initialization, and cannot recover from tracking failures.
- 3D Human Pose Estimation in Image Sequences: Candidate-linking methods detect poses independently and connect them across frames using temporal consistency.
- 3D Human Pose Estimation in Image Sequences: Spatiotemporal features have been underused for 3D pose estimation, while trajectory-based alternatives can be very computationally costly.
3. Method
The method predicts the central-frame 3D pose directly from motion-compensated spatiotemporal volumes. It aligns subjects across frames, extracts multiscale 3D HOG features, and regresses the pose while avoiding explicit per-frame pose tracking.
- Pose Regression: Direct regression uses a CNN-based approach to estimate the central-frame pose from the aligned volume, rather than linking independently estimated poses afterward.The pipeline detects people, aligns image windows, builds an RSTV, extracts 3D HOG features, and regresses the pose.
- Pose Regression: The model maps a spatiotemporal feature vector to the 3D pose in the central frame.Training uses consecutive frame blocks, with the best results reported for T = 24 to 48 images, or 0.5 to 1 second at 50 fps.
- Spatiotemporal Features: The feature vector concatenates multiscale 3D HOG descriptors that encode appearance and motion across the volume.The descriptor uses 2×2, 4×4, and 8×8 spatial cells with temporal cells of 4 frames for 50 fps videos.
- Spatiotemporal Features: The approach avoids explicitly tracking body pose in each frame, which remains vulnerable to ambiguities from 3D-to-2D projection.Instead, temporal information is encoded through spatiotemporal features before regression.
- Spatiotemporal Features: Direct 3D CNNs on pixel-intensity volumes did not notably improve performance in the authors’ experiments.The authors attribute this observation to optimization difficulty from model complexity and large input dimensionality.
- Motion Compensation: The method assumes temporal bins correspond to specific body parts, making accurate frame-to-frame centering necessary for representative 3D HOG features.DPM detections may be misaligned, motivating the learned motion-compensation step.
- Motion Compensation: Motion compensation stabilizes the extracted features by making body parts covariant with 3D HOG cells across frames.Without compensation, gradients disperse across the region of interest, reducing feature stability.
- Motion Compensation: Bounding boxes from consecutive frames are motion-compensated so the subject remains centered before forming the spatiotemporal volume.Two CNN regressors handle large and small shifts iteratively, producing a rectified spatiotemporal volume.
4. Results
Across Human3.6m, HumanEva-I/II, and KTH Multiview Football II, the method is evaluated against diverse state-of-the-art baselines using motion-compensated spatiotemporal features. Results show consistent gains, with motion compensation and an appropriately sized temporal window improving accuracy.
- Datasets and protocol: The evaluation covers Human3.6m, HumanEva-I/II, and KTH Multiview Football II, including studio and non-studio settings.Human3.6m includes varied subjects, actions, viewpoints, body shapes, clothing, poses, and viewing angles; Football II tests a moving-camera environment.
- Human3.6m results: 23% mean error reduction is reported against eχ2-HOG+KDE on all Human3.6m actions.The method also exceeds CNN-Regression by more than 5% on every action with reported accuracy numbers.
- Human3.6m results: RSTV+DN achieves the best accuracy on 12 of 15 Human3.6m actions and on average across all actions.RSTV+KDE performs better for Buying, Sitting, and Sitting Down, while gains are especially marked for Walking and Eating.
- Ablation studies: Motion compensation significantly improves spatiotemporal-volume accuracy over STV, and CNN-based compensation outperforms optical-flow compensation.Even without compensation, regression on spatiotemporal volumes is more accurate than the method of.
- Ablation studies: 24–48 frames, corresponding to 0.5–1 second at 50 fps, gives the best temporal-window results.A 12-frame window already outperforms, while smaller windows lack information and overly large windows risk overfitting.
- HumanEva results: On HumanEva-I, the method outperforms state-of-the-art approaches on Walking and Boxing sequences.Because HumanEva training data is small, the experiments use RSTV+KDE rather than RSTV+DN.
- HumanEva results: On HumanEva-II, training only on HumanEva-I still yields state-of-the-art performance across different camera views.The method uses less additional training data than the comparison framework described in.
- KTH Multiview Football results: On KTH Multiview Football II, the monocular method outperforms baselines using both cameras.The paper attributes this to simultaneous use of appearance and motion from rectified spatiotemporal volumes.
5. Conclusion
Early motion integration through rectified spatiotemporal volumes improves 3D pose estimation by disambiguating mirroring and self-occlusion. The framework also extends to other articulated motions.
- Early motion integration improves performance over linking pose estimates after individual-frame prediction.
- Rectified spatiotemporal volumes combine appearance and motion cues to disambiguate poses involving mirroring and self-occlusion.
- The framework substantially increases accuracy over state-of-the-art methods on several 3D human pose benchmarks.
- The proposed framework is generic and could be used for other kinds of articulated motions.