Source-linked AI summary
Motion Guided 3D Pose Estimation from Videos
Jingbo Wang, Sijie Yan, Yuanjun Xiong, Dahua Lin
TL;DR
Monocular 3D pose estimation from 2D pose is ambiguous because depth is missing and conventional losses overlook motion structure. The paper introduces motion loss with pairwise motion encoding and UGCN, achieving state-of-the-art performance on two large datasets while modeling temporal dependencies.
Problem
Monocular 3D pose estimation is ill-posed because 2D projections lack depth, and Minkowski Distance does not capture temporal structure in pose sequences.
Method
Motion loss compares predicted and ground-truth locations in pairwise motion-encoding space, while UGCN captures short-term and long-term temporal dependencies.
Results
UGCN with motion loss achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP, with multiscale motion loss reducing MPJPE by 1.5mm from 27.1 to 25.6.
Takeaways & Limitations
Motion-based supervision improves 3D pose accuracy and supports modeling motion across varied temporal scales.
Abstract
from arXiv · showhide
We propose a new loss function, called motion loss, for the problem of monocular 3D Human pose estimation from 2D pose. In computing motion loss, a simple yet effective representation for keypoint motion, called pairwise motion encoding, is introduced. We design a new graph convolutional network architecture, U-shaped GCN (UGCN). It captures both short-term and long-term motion information to fully leverage the additional supervision from the motion loss. We experiment training UGCN with the motion loss on two large scale benchmarks: Human3.6M and MPI-INF-3DHP. Our model surpasses other state-of-the-art models by a large margin. It also demonstrates strong capacity in producing smooth 3D sequences and recovering keypoint motion.
1 Introduction
Monocular 3D pose estimation is ill-posed because 2D projections lack depth, while conventional per-location losses overlook temporal structure. The paper addresses this with motion loss and UGCN, improving motion-aware 3D pose estimation.
- 2D projections lack depth information, making monocular 3D human pose estimation ambiguous.
- Minkowski Distance evaluates pose locations independently and does not measure temporal-structure similarity between estimated and ground-truth sequences.
- Motion loss supervises both 3D keypoint locations and motion trajectories using differentiable pairwise motion encoding across multiple time intervals.
- UGCN uses a U-shaped temporal structure to capture both short-term and long-term dependencies needed to characterize keypoint motion.
- Motion loss improves 3D pose estimation accuracy, and UGCN with motion loss surpasses current state-of-the-art models on Human3.6M and MPI-INF-3DHP.
2 Related work
Prior work includes direct image-to-3D estimation, two-stage 2D-to-3D lifting, and graph convolution over spatial-temporal skeleton graphs. The proposed approach builds on the two-stage pipeline and graph-based modeling.
- 3D pose estimators have been developed using handcrafted features and deep neural networks.
- Direct methods predict 3D poses from 2D images using approaches including joint regression, spatial-temporal features, voxel likelihoods, and latent pose representations.
- Two-stage methods first estimate 2D pose sequences from video and then lift them to 3D using fully connected, temporal-convolutional, or graph-convolutional models.
- The proposed method also uses a two-stage pipeline, benefiting from reduced data dimensionality that makes long-term video-based 3D pose estimation possible.
- Spatial-temporal graph convolution has improved human understanding tasks including action recognition, pose tracking, and motion synthesis.
3 Approach
The approach combines multiscale pairwise motion supervision with a U-shaped graph convolutional network that models spatial structure and both short- and long-range temporal information. The resulting end-to-end model reconstructs 3D keypoint locations while preserving motion-related information.
- 3 Approach: The model predicts 3D coordinates from 2D pose sequences by structuring keypoints as a spatial-temporal graph and applying UGCN end to end.The graph connects corresponding joints across adjacent frames and bone endpoints within each frame, jointly representing temporal dependencies and spatial configuration.
- 3.1 Motion Loss: Pairwise motion encoding represents trajectories through differentiable operations applied to coordinate pairs across multiple temporal intervals.The encoding is designed to be differentiable, nonindependent across time, and multiscale; experiments consider subtraction, inner-product, and cross-product operators.
- 3.1 Motion Loss: Motion loss computes reconstruction error in the encoded motion space across joints, moments, and multiple time intervals, while a traditional reconstruction term preserves absolute location information.The combined objective is trained end to end, with λ balancing the motion and traditional reconstruction objectives.
- 3.2 U-shaped Graph Convolutional Networks: Spatial-temporal graph convolution aggregates temporal features and skeleton-structure information using separate temporal and spatial operations with neighborhood subsets.Neighborhood labels are defined by hop distance from the central hip, enabling distinct treatment of central and limbic joints.
- 3.2 U-shaped Graph Convolutional Networks: UGCN uses downsampling, upsampling, and merging stages to combine long-range temporal aggregation, recovered resolution, low-level shortcuts, and multiscale feature fusion.Temporal pooling expands the receptive field, upsampling restores temporal resolution, and features from the downsampling stage provide low-level information to the upsampling branch.
4 Experiments
Experiments evaluate motion loss and UGCN through ablations and comparisons on Human3.6M and MPI-INF-3DHP. The results show gains from multiscale motion supervision and U-shaped temporal modeling, with strong benchmark performance and smoother predicted sequences.
- Experimental setup: Experiments use Human3.6M and MPI-INF-3DHP, with ablations using ground-truth 2D poses to isolate the proposed components.Human3.6M provides 3D ground truth and camera parameters, though the proposed approach does not use camera parameters; MPI-INF-3DHP adds PCK and AUC evaluation.
- Motion-loss ablations: 4.9mm lower MPJPE, from 32.0 to 27.1, is achieved with cross-product encoding at a 12-frame interval.Motion loss improves performance across a broad interval range of 2–36 frames, including long intervals such as 24 frames.
- Motion-loss ablations: 1.5mm lower MPJPE, from 27.1 to 25.6, results when four selected intervals are combined into multiscale motion loss.The intervals are selected from the single-scale experiments, using cross-product encoding.
- UGCN ablations: The U-shaped structure improves MPJPE by 6.6mm without motion loss and by 9.8mm with motion-loss supervision.The ablation removes and then restores paired downsampling and upsampling operations, supporting the role of long-term perception.
- UGCN ablations: Removing the merging stage increases error by 2.8mm with motion loss versus 0.5mm without it, indicating the importance of multiscale temporal information.The spatial configuration itself contributes a 7.2mm improvement.
- Benchmark comparison: The method achieves 45.6 MPJPE under Human3.6M Protocol 1, 35.5 P-MPJPE under Protocol 2, and 86.9 PCK, 62.1 AUC, and 68.1 MPJPE on MPI-INF-3DHP.On Human3.6M, these results surpass the reported baselines; HR-Net input further improves MPJPE by roughly 3mm, while motion loss reduces MPJVE by 32%, from 3.4mm to 2.3mm.
5 Conclusion
The paper introduces motion loss and an U-shaped model to capture temporal dependencies in 3D pose sequences. On two large datasets, UGCN with motion loss achieves state-of-the-art performance.
- Motion loss explicitly incorporates motion modeling into the learning objective.
- The U-shaped model captures both short-term and long-term temporal dependencies in pose sequences.
- UGCN combined with motion loss achieves state-of-the-art performance on two large datasets.
- Motion loss may inspire applications in action forecasting, action generation, and pose tracking.
Impact of 2D Pose Estimators
The study examines how input 2D pose quality affects monocular 3D pose estimation. It reports a near-linear relationship between 2D estimator quality and 3D MPJPE.
- A near-linear relationship is observed between the MPJPE of 3D poses and the quality of input 2D poses.
- HR-Net predicted 2D poses yield lower MPJPE than CPN predicted 2D poses.
- The analysis simulates multiple 2D pose estimators by weighted addition of predicted 2D poses and ground truth.
Visual Results
The paper visualizes 3D pose sequences estimated by UGCN on MPI-INF-3DHP and Human3.6M. Additional supplementary videos show motion-loss effects and comparisons with previous methods.
- Supplementary visualizations include the impacts of motion loss and comparisons with previous works.
- Figure 7 presents UGCN-estimated 3D pose sequences from MPI-INF-3DHP and Human3.6M.