Source-linked AI summary
Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video
Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee
TL;DR
Video-based 3D human mesh recovery still struggles with temporally inconsistent motion because of dependence on the current frame’s static feature. TCMR removes that dependence by forecasting and integrating temporal features from past, future, and all frames, achieving better temporal consistency and per-frame pose and shape accuracy than prior video methods.
Problem
Video-based 3D human pose and shape estimation remains temporally inconsistent because methods strongly depend on the current frame’s static feature.
Method
TCMR removes the static-temporal residual connection and uses PoseForecast to obtain current-target temporal features from past and future frames for integration.
Results
TCMR outperforms previous video-based methods in temporal consistency while achieving better per-frame 3D pose and shape accuracy.
Takeaways & Limitations
Forecasting current temporal features from past and future frames reduces dependence on the current static feature and supports temporally consistent 3D human motion.
Abstract
from arXiv · showhide
Despite the recent success of single image-based 3D human pose and shape estimation methods, recovering temporally consistent and smooth 3D human motion from a video is still challenging. Several video-based methods have been proposed; however, they fail to resolve the single image-based methods' temporal inconsistency issue due to a strong dependency on a static feature of the current frame. In this regard, we present a temporally consistent mesh recovery system (TCMR). It effectively focuses on the past and future frames' temporal information without being dominated by the current static feature. Our TCMR significantly outperforms previous video-based methods in temporal consistency with better per-frame 3D pose and shape accuracy. We also release the codes. For the demo video, see https://youtu.be/WB3nTnSQDII. For the codes, see https://github.com/hongsukchoi/TCMR_RELEASE.
1. Introduction
Monocular 3D human pose and shape estimation remains difficult, and applying single-image methods frame by frame produces temporally inconsistent motion. TCMR addresses this by emphasizing past and future temporal information, improving both motion consistency and per-frame accuracy.
- Monocular 3D human pose and shape estimation remains challenging because of depth ambiguity, limited training data, and complex articulations.
- Single-image methods can produce reasonable static outputs but temporally inconsistent, unsmooth video motion from inconsistent errors across consecutive frames.
- Existing video methods improve per-frame pose accuracy and smoothness, yet still exhibit temporal inconsistency because they depend strongly on the current frame’s static feature.
- TCMR removes the static-temporal residual connection and uses PoseForecast to predict the current pose from past and future frames without the current frame.
- TCMR improves temporal consistency and per-frame 3D pose and shape accuracy, outperforming previous video-based methods.
2. Related works
Prior work spans single-image mesh regression, video-based temporal modeling, and pose forecasting. TCMR differs by forecasting useful temporal features without relying on the current static feature.
- Single-image methods commonly regress parameters of the predefined SMPL mesh model, while model-free methods estimate human shape directly.
- Video-based methods extend image models with temporal encoders, but reported temporal-consistency evaluation has historically been limited.
- MEVA balances per-frame pose accuracy and temporal smoothness, yet qualitative results still show unsmooth motion linked to current static-feature dependence.
- Earlier pose-forecasting methods predict future or past poses, whereas TCMR forecasts temporal features to learn useful motion information independent of the current static feature.
3. TCMR
TCMR extracts static features from video frames, encodes information from all, past, and future frames, and integrates these temporal features for mesh estimation. During inference, the integrated feature is regressed into the final SMPL mesh parameters.
- Temporal encoding from all frames: A shared pretrained ResNet extracts a 2048-dimensional static feature from each of T input RGB frames.
- Temporal encoding from all frames: A bidirectional GRU encodes temporal information from all input frames without a residual connection from the current static feature.
- Temporal encoding by PoseForecast: PoseForecast uses separate GRUs to forecast current-target temporal features from past and future frames.
- Temporal feature integration: TCMR transforms and concatenates the all-frame, past, and future temporal features, then predicts attention values for their integration.
- Temporal feature integration: The shared SMPL regressor produces parameters from each temporal feature during training, while inference uses the integrated feature to produce Θint and the final mesh.
- Temporal feature integration: Training supervises all three outputs with current-frame ground truth using SMPL-parameter and 2D/3D joint-coordinate losses.
4. Implementation details
The implementation follows a fixed video-input and training setup based on prior work, with pretrained initialization, cropped 224×224 images, and Adam optimization. Occlusion augmentation reduces pose and acceleration errors by approximately 1mm.
- The model uses 16 frames at 25–30 frames per second, pretrained SPIN initialization, and ground-truth human-region crops resized to 224×224.
- Training uses Adam with mini-batches of 32, an initial learning rate of 5−5, and 30 epochs on one NVIDIA RTX 2080Ti GPU.
- Occlusion augmentation reduces both pose and acceleration errors by approximately 1mm.
- Static features are precomputed from cropped images to save training time and memory, and 6D rotations are converted to axis-angle form.
5. Experiment
Experiments evaluate TCMR’s per-frame accuracy and temporal consistency through architectural ablations and comparisons with prior video- and image-based methods. Forecasting temporal features from past and future frames reduces reliance on the current static feature and produces smoother motion.
- Evaluation metrics: Per-frame accuracy is measured with MPJPE, PA-MPJPE, and MPVPE in millimeters, with PA-MPJPE serving as the main accuracy metric.PA-MPJPE removes the effect of output scale ambiguity after root-joint alignment.
- Temporal architecture ablation: Temporal encoding that uses all frames with the current frame may be suboptimal, whereas forecasting current temporal features from past and future frames consistently reduces acceleration error by over 11%.The ablation uses the VIBE baseline architecture and examines how its components affect dependence on current static features.
- Temporal architecture ablation: Adding current frames to PoseForecast increases acceleration error by 33%, supporting temporal features that remain free from the current static feature.The comparison directly tests PoseForecast variants with and without the current frame.
- Comparison with prior methods: TCMR shows lower acceleration errors over time than MEVA and VIBE, while previous methods exhibit pronounced spikes associated with unstable motion estimation.The reported comparison indicates temporally consistent 3D motion without relying on post-processing.
- Comparison with prior methods: TCMR outperforms previous video-based methods across all evaluated benchmarks in both per-frame 3D pose accuracy and temporal consistency.The comparison follows shared training-data conditions for methods other than HMMR and uses the protocols’ input-frame counts.
- Comparison with prior methods: On 3DPW and MPI-INF-3DHP, TCMR surpasses prior methods in PA-MPJPE and temporal consistency, with especially large temporal-consistency gains over single-image methods.The paper also reports more than three times better performance than SPIN and Pose2Mesh in the cited comparison, while VIBE reduces SPIN’s acceleration error by 9%.
6. Conclusion
The conclusion presents TCMR as a video-based 3D human mesh recovery system that addresses temporal inconsistency by reducing reliance on the current frame. It combines removal of the static-temporal residual connection with PoseForecast from past and future frames.
- 6. Conclusion: TCMR estimates 3D human meshes from RGB videos while targeting temporally consistent motion.The system is designed for video-based 3D human pose and shape recovery.
- 6. Conclusion: Removing the residual connection and forecasting current temporal features from past and future frames addresses dependence on the current static feature.The conclusion identifies this dependence as the source of temporal inconsistency in previous video-based methods.
- 6. Conclusion: Compared with previous video-based methods, TCMR provides more temporally consistent 3D motion and more accurate per-frame 3D pose.
7. More qualitative results
Additional qualitative results show TCMR on fast, diverse in-the-wild videos and compare its motion outputs with VIBE and MEVA under a fixed-camera presentation.
- 7. More qualitative results: The qualitative results include TCMR outputs on fast and diverse in-the-wild videos from 3DPW.Outputs are also rendered from the opposite view.
- 7. More qualitative results: A fixed camera and plain background are used to compare TCMR, VIBE, and MEVA for temporal consistency and motion smoothness.
8. Human evaluation.
A human study compared the perceived realism of TCMR, MEVA, and VIBE outputs across 20 3DPW sequences. TCMR received the largest share of votes.
- 8. Human evaluation: 69% of participants selected TCMR as producing the most realistic motion, compared with 26% for MEVA and 5% for VIBE.The survey covered 20 sequences from the 3DPW validation and test sets and 50 participants.
9. Attention values in feature integration.
TCMR assigns greater weight to past and future temporal features than to the current temporal feature, with attention varying by motion difficulty.
- Past and future temporal features receive more weight than the current temporal feature during integration.Each attention value varies within ±20%.
- The current temporal feature’s attention remains at least 0.4 during slow-motion walking.
- During fast basketball movements with complex poses, the current temporal feature’s attention generally stays below 0.4.
10. Datasets
The evaluation spans in-the-wild, indoor, 3D, and 2D human video benchmarks, while experiments examine pretrained static features and report broad performance claims.
- Datasets: 3DPW contains 60 in-the-wild videos and 51K frames with 3D pose and shape annotations, captured by phone at 30 fps.Its official split contains 24 training, 12 validation, and 24 test videos.
- Datasets: Human3.6M provides 3.6M indoor video frames across 15 action categories, with training on five subjects and testing on two.
- Datasets: MPI-INF-3DHP includes indoor and outdoor actions, with positional errors measured on valid frames and acceleration error computed on all frames.
- Datasets: InstaVariety contains 28K Instagram videos, while Penn Action contains 2.3K sports-action sequences and 77K annotated frames.
- Static-feature evaluation: Models trained without SPIN-pretrained ResNet features show very high per-frame 3D pose errors.Table 7 compares these models with TCMR and a baseline using the same SPIN-pretrained SMPL parameter regressor.
- Findings: TCMR significantly improves temporal consistency while preserving per-frame 3D pose accuracy using ResNet static features pretrained on large-scale in-the-wild 2D pose data.
12. Effect of input fps
Reducing input frame rate leaves accuracy relatively stable but worsens acceleration error, while TCMR remains substantially more temporally consistent than smoothed Pose2Mesh.
- The acceleration error doubles when input fps is reduced by half, while accuracy remains relatively unchanged.This suggests temporally dense input is important for output temporal consistency.
- TCMR can fix invalid poses using relatively sparse temporal information despite the increased acceleration error at lower fps.
- TCMR outperforms temporally smoothed Pose2Mesh by nearly twice in temporal consistency without post-processing.Smoothed Pose2Mesh reports PA-MPJPE 58.6, MPJPE 89.6, and acceleration error 12.9 on 3DPW.