Source-linked AI summary
3D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Training
Yu Cheng, Bo Yang, Bo Wang, Robby T. Tan
TL;DR
Monocular-video 3D pose estimation remains difficult when person scale, motion speed, or occlusion differs from training conditions. The paper introduces a multi-scale spatio-temporal network with sequence-level pose discrimination and explicit occlusion training, and reports improvements across public datasets.
Problem
Existing monocular-video 3D pose estimators are limited by varying person scales and motion speeds, and many are not explicitly trained for severe occlusion.
Method
The method combines multi-scale spatial and temporal features, a spatio-temporal KCS discriminator, and diverse keypoint or frame masking for occlusion augmentation.
Results
The method reduces Human3.6M MPJPE by 2.8mm and P-MPJPE by 2.1mm, improves HumanEva-I by 0.8mm, and raises MPI-INF-3DHP 3D PCK by 1.6%.
Takeaways & Limitations
The method handles varied motion speeds and occlusion types, with ablations supporting the effectiveness of its individual components.
Abstract
from arXiv · showhide
Estimating 3D poses from a monocular video is still a challenging task, despite the significant progress that has been made in recent years. Generally, the performance of existing methods drops when the target person is too small/large, or the motion is too fast/slow relative to the scale and speed of the training data. Moreover, to our knowledge, many of these methods are not designed or trained under severe occlusion explicitly, making their performance on handling occlusion compromised. Addressing these problems, we introduce a spatio-temporal network for robust 3D human pose estimation. As humans in videos may appear in different scales and have various motion speeds, we apply multi-scale spatial features for 2D joints or keypoints prediction in each individual frame, and multi-stride temporal convolutional net-works (TCNs) to estimate 3D joints or keypoints. Furthermore, we design a spatio-temporal discriminator based on body structures as well as limb motions to assess whether the predicted pose forms a valid pose and a valid movement. During training, we explicitly mask out some keypoints to simulate various occlusion cases, from minor to severe occlusion, so that our network can learn better and becomes robust to various degrees of occlusion. As there are limited 3D ground-truth data, we further utilize 2D video data to inject a semi-supervised learning capability to our network. Experiments on public datasets validate the effectiveness of our method, and our ablation studies show the strengths of our networkś individual submodules.
Introduction
The paper targets monocular-video 3D pose estimation under varying person scales, motion speeds, and occlusions. It combines multi-scale spatial-temporal features, spatio-temporal pose validation, and occlusion augmentation.
- Introduction: Existing methods underuse video spatial-temporal information, limiting robustness to different target sizes and motion speeds.
- Introduction: Multi-scale spatial features and multiple temporal TCN strides combine richer frame representations with motion information for 3D pose prediction.
- Introduction: A temporal KCS discriminator evaluates whether estimated pose sequences form valid movements, extending single-frame pose validity checks.
- Introduction: Training masks keypoints or frames to cover partial and total occlusions across individual frames or sequences.
- Introduction: The framework’s contributions are multi-scale spatial-temporal features, spatio-temporal pose regularization, and diverse occlusion augmentation.
Related Works
Prior work improves 2D and 3D pose estimation through spatial and temporal modeling, pose regularization, and occlusion-aware prediction. However, fixed temporal scales and frame-wise pose validation leave important sequence-level challenges.
- Related Works: The field also explores human dynamics, pose regularization, and semi-supervised or self-supervised learning alongside improved spatial-temporal representations.
- Related Works: Spatial feature aggregation and temporal modeling improve pose estimation, but fixed temporal scales limit performance under different motion speeds.
- Related Works: Pose discriminators assess whether individual estimated 3D poses match ground-truth pose distributions, while sequence validity remains insufficiently addressed.
- Related Works: Occlusion methods recover missing keypoints from spatial or temporal context, while human-dynamics methods predict future poses from available frames.
Methodology
The method combines multi-scale spatial and temporal processing, spatio-temporal pose discrimination, multi-view and 2D supervision, and occlusion-focused augmentation for monocular 3D pose estimation.
- The pipeline detects and tracks each person, normalizes their image patches, and estimates poses individually from the input video.
- Multi-Scale Features for Pose Estimation: HRNet produces K joint heat maps using repeated multi-scale feature fusion, while an embedding network preserves information from complete heat maps.The embedding uses the heat maps rather than only their peaks.
- Multi-Scale Features for Pose Estimation: Multi-stride TCNs with temporal strides 1, 2, 3, 5, and 7 combine motion scales, improving predictions for fast-motion clips over single-scale TCNs.
- Spatio-Temporal KCS Pose Discriminator: A spatio-temporal discriminator combines spatial KCS, temporal KCS, and predicted coordinates to regularize both pose validity and transitions across frames.Temporal KCS represents bone-length and inter-bone-angle changes between frames.
- Data Augmentation for Occlusions: Occlusion augmentation masks complete frames, selected keypoints, or spatial areas, while additional noise and point perturbations simulate imperfect detections.When masks occupy the receptive field’s end, the framework reduces to future-pose estimation without future observations; otherwise it uses broader temporal context.
Experiments
Experiments show that the framework improves 3D pose estimation across standard indoor, cross-dataset, outdoor, and human-dynamics evaluations, while ablations support its multi-scale and spatio-temporal components.
- Ablation Studies: Ablations show steadily improving performance as modules are added, with the largest gains from multi-stride TCN, spatial KCS, and temporal KCS.Temporal KCS further improves over spatial KCS, supporting sequence-level validity checking beyond individual-frame pose constraints.
- Quantitative Results: 2.8mm lower MPJPE and 2.1mm lower P-MPJPE than previous work on Human3.6M, corresponding to 6.5% and 6.4% error reductions.Improvements exceed 5mm for photo capturing and sitting down, where occlusion occurs frequently.
- Human Dynamics: The framework achieves similar performance to state of the art for human dynamics despite using video images rather than past 3D ground-truth keypoints as input.This evaluation tests future 3D information prediction after first estimating keypoints from images.
- Cross-Dataset Results: 1.6% higher 3D PCK on MPI-INF-3DHP without retraining or fine-tuning on that dataset.The evaluation uses only keypoints overlapping with Human3.6M and considers errors under 150mm correct.
- Cross-Dataset Results: On HumanEva-I, the method improves average error by 0.8mm, a 5.6% error reduction.The model trained on Human3.6M also improves performance on MPI-INF-3DHP without dataset-specific retraining or fine-tuning.
- Outdoor Evaluation: 71.8 P-MPJPE on the 3DPW test set outperforms the 157.0 and 80.1 results reported by Martinez et al. and Kanazawa et al.The model is evaluated outdoors without training on 3DPW.
Conclusion
The method combines multi-scale temporal features, a spatio-temporal KCS discriminator, and occlusion augmentation for robust video pose estimation. It outperforms state-of-the-art methods across four public 3D pose datasets and may extend to pose forecasting.
- The method achieves superior performance over state-of-the-art 3D pose estimation methods across four public datasets.Evaluations use commonly used protocols, with ablation studies assessing each component.
- Its three main components are multi-scale temporal features, a spatio-temporal KCS pose discriminator, and occlusion data augmentation.Together, these components target motion-speed variation, pose-sequence validity, and different occlusion cases.
- Comparison with human-dynamics methods indicates versatility and potential applicability to other pose tasks, including pose forecasting.