Source-linked AI summary
3D Human Motion Estimation via Motion Compression and Refinement
Zhengyi Luo, S. Alireza Golestaneh, Kris M. Kitani
TL;DR
3D human motion estimation must preserve both spatial accuracy and temporal smoothness while retaining person-specific details. MEVA uses compressed general motion followed by image-based residual refinement, producing smooth and accurate estimates, including a 54.3% reduction in acceleration error on 3DPW.
Problem
3D pose estimates can remain temporally jittery because accuracy-focused metrics such as MPJPE do not account for motion smoothness or person-specific motion details.
Method
MEVA first estimates coarse motion with a VAE-based Variational Motion Estimator, then adds person-specific details through image-based Motion Refinement Regression.
Results
The method achieves smooth and accurate motion, reducing acceleration error by 54.3% on 3DPW while attaining state-of-the-art MPJPE results.
Takeaways & Limitations
Separating general smooth motion from person-specific refinement preserves motion details while improving temporal smoothness without sacrificing pose accuracy.
Takeaways & Limitations
The sliding-window design can produce inconsistencies at three-second intervals, and occlusions can cause detailed motion to be miscaptured.
Abstract
from arXiv · showhide
We develop a technique for generating smooth and accurate 3D human pose and motion estimates from RGB video sequences. Our method, which we call Motion Estimation via Variational Autoencoder (MEVA), decomposes a temporal sequence of human motion into a smooth motion representation using auto-encoder-based motion compression and a residual representation learned through motion refinement. This two-step encoding of human motion captures human motion in two stages: a general human motion estimation step that captures the coarse overall motion, and a residual estimation that adds back person-specific motion details. Experiments show that our method produces both smooth and accurate 3D human pose and motion estimates.
1 Introduction
The paper addresses the tension between accurate but temporally jittery 3D pose estimates and smooth motion. MEVA separates coarse general motion estimation from person-specific refinement to preserve both properties.
- Approach: MEVA decomposes motion into a coarse general component and a person-specific component, enabling more accurate and smooth estimates.
- Motivation: 3D pose estimation has improved in accuracy, but MPJPE does not measure temporal consistency, allowing estimates to jitter across frames.The resulting motion can appear unnatural even when spatial joint errors are low.
- Motivation: Prior approaches improve smoothness through motion datasets, adversarial losses, or temporal constraints, but loss-based priors make balancing smoothness and accuracy difficult.
- Approach: The method first compresses motion with a VAE-based Variational Motion Estimator, then refines the pose using image evidence through a Motion Refinement Regressor.
- Results: On 3DPW, the method reduces acceleration error by 54.3% while achieving state-of-the-art MPJPE results.
2 Related Works
Related work covers image- and video-based pose recovery, temporal models, motion priors, and compact latent representations for human motion.
- Pose and motion recovery: Human motion recovery is treated as a sequence of human poses, connecting single-image and video-based human shape and pose estimation.
- Temporal estimation: Temporal methods lift 2D joint predictions to 3D or predict 3D joints directly, using LSTMs, temporal convolutions, fully connected layers, or temporal filters.
- Motion priors: Prerecorded motion sequences have been used to quantify unnatural motion, support 3D tracking, and constrain recovered poses with adversarial discriminators.
- Latent motion representations: Motion compression methods learn compact latent representations for generation, cross-modal generation, and trajectory forecasting using VAEs, generative flows, or GANs.
3 Approach
MEVA estimates coherent 3D human motion from video by extracting temporal features, learning coarse motion in a VAE-based latent space, and refining it with person-specific residual details. The approach uses SMPL motion representations and augmentation to improve the learned motion subspace’s generalization.
- Overall framework: MEVA extracts temporally correlated video features, estimates coarse motion with a variational motion estimator, and adds fine details using a motion residual regressor.The framework’s three stages are STE, VME, and MRR.
- Problem formulation: The task is to recover coherent SMPL pose sequences from video, with SMPL parameters representing human shape and pose and supporting 3D mesh and joint recovery.SMPL provides a representation that can be converted to 3D joint positions and human meshes.
- Spatio-temporal feature extraction: MEVA uses a GRU-based spatio-temporal feature extractor because human motion is temporally correlated and past movement can cue future motion.The extractor encodes video frames into temporally correlated features for sequence modeling.
- Variational motion estimation: The VAE learns a smooth latent human-motion subspace by mapping motion sequences to latent codes under a Gaussian prior and decoding them back into motion sequences.The encoder processes sequences of SMPL pose parameters, while the decoder reconstructs motion from the latent code.
- Motion-data augmentation: MEVA augments motion data through speed changes, left-right flipping, and random root rotations to produce varied but plausible sequences and improve generalization to unseen motion.The augmentation scheme addresses poor generalizability of the VAE on unseen sequences.
- Motion refinement: The MRR initializes regression from VME’s coarse poses rather than the mean pose, so it makes small refinements that restore person-specific motion details.The regressor jointly refines motion and human shape estimates using temporally encoded visual features.
4 Experiments
The experiments evaluate MEVA on multiple motion datasets using accuracy and smoothness metrics. Results show that the motion VAE generalizes to unseen sequences, while the full method improves smoothness without sacrificing comparable positional accuracy.
- Evaluation setup: MEVA is evaluated on MPI-INF-3DHP, 3DPW, and Human3.6M using MPJPE, PA-MPJPE, and acceleration error.MPJPE and PA-MPJPE measure positional discrepancy, while acceleration error measures motion smoothness.
- Motion VAE generalization: The motion VAE generalizes to unseen 3DPW sequences and represents the human motion space with reasonable reconstruction quality.The evaluation uses unseen sequences from different 3DPW splits.
- Overall results: MEVA achieves state-of-the-art video motion estimation results across all three test datasets.The comparison reports positional accuracy and acceleration error against state-of-the-art methods.
- Motion quality: The method preserves high joint position accuracy while reducing acceleration error, producing smoother and more natural motion.The reported result characterizes the improvement as increased smoothness without sacrificing accuracy.
4.3 Ablation Experiments
The ablations show that data augmentation improves motion-VAE generalizability, while MEVA’s coarse estimator captures smooth motion and refinement restores fine details; pretrained VAE latents help avoid occasional invalid poses.
- Data augmentation: Random root rotation is essential for motion-VAE generalizability, while frame-rate variation and left-right flipping also provide significant gains.Using all three augmentation techniques produces the best-performing motion VAE on unseen 3DPW sequences.
- Motion decomposition: The VME-only model captures coarse, smooth motion and achieves low-acceleration performance similar to HMMR.The coarse/fine decomposition is visualized by showing VME output before MRR adds fine details.
- Pretrained VAE: Removing the pretrained VAE preserves relatively strong accuracy and smoothness but produces a few kinematically invalid poses during sequences.The corresponding qualitative ablation shows unnatural poses in the first few frames.
5 Conclusion
The paper concludes that smooth and accurate 3D motion requires combining a compressive model for general smooth motion with image-based refinement for person-specific details. MEVA implements this through a two-stage VAE and regression framework and demonstrates smooth, accurate estimates.
- Conclusion: MEVA combines a motion-compression model encoding general smooth motion with image-based regression that captures person-specific motion.The two components address complementary aspects of human motion estimation.
- Conclusion: The two-stage model first trains a VAE for coarse human motion and then learns a refinement module to retain motion details absent from the general model.This design explicitly separates coarse motion modeling from person-specific refinement.
- Conclusion: Comprehensive experiments demonstrate that the method produces both smooth and accurate 3D human motion.
Supplementary Material
The supplementary material contains qualitative results, complementary ablations, failure-mode discussion, and implementation details.
- Supplementary Material: The supplement is organized into qualitative MEVA results, complementary ablation studies, failure modes, and implementation details.
1 Qualitative Results
The supplementary video presents MEVA’s coarse-to-fine decomposition, compares it qualitatively with VIBE, and covers implementation details.
- Qualitative Results: The supplementary video demonstrates the two-stage decomposition of coarse and fine motion from a video sequence.
- Qualitative Results: The video qualitatively compares MEVA with VIBE and reports smoother, more natural, and accurate motion estimation for MEVA.
- Qualitative Results: The supplementary video also discusses implementation details of the method.
2 Additional Ablation Studies
Additional studies examine postprocessing, temporal-window length, and the contributions of MEVA’s components. They show that refinement helps avoid the overly smoothed motion produced by coarse estimation alone, while average filtering remains complementary.
- Comparison with Average Filtering: Average filtering reduces acceleration error for both VIBE and MEVA while slightly affecting accuracy.The paper treats postprocessing as complementary to feed-forward methods.
- Temporal Window: MEVA uses a 90-frame temporal window, compared with 20 frames for HMMR and 16 frames for VIBE.The longer window is part of the comparison with prior methods.
- Temporal Window: With the same 90-frame temporal window, MEVA performs better than VIBE on all three metrics and retains a significant acceleration-error advantage.Retraining VIBE with the longer window produces only a slight improvement over its original setting.
- Component Ablation: Using only VME produces overly smoothed motion and higher acceleration error because it estimates coarse motion without refinement.Removing MRR reduces the system to a single coarse-motion stage.
3 Failure Modes
MEVA has failure modes related to sliding-window transitions, visual occlusion, and the body model’s scope. These limitations can disrupt temporal consistency, omit detailed motion, or exclude hands and facial expressions.
- Sliding window processing: Sliding-window processing can create inconsistency at roughly 3-second intervals because neighboring 90-frame windows may estimate different coarse motions.MRR may not fully adjust the transition, and each window has separate STE features.
- Occluded body parts: Occluded body parts remain challenging because missing visual evidence forces MEVA to rely on coarse motion and can miss detailed movement.The limitation arises when image evidence is unavailable during occlusion.
- Missing hands and face movement: Methods using the original SMPL model do not capture hand movements or facial expressions because SMPL lacks hand and face joints.The paper identifies capturing these movements as an important direction for future work.
4 Implementation Details
MEVA combines recurrent motion compression, temporally correlated video features, and iterative pose refinement. Its motion VAE encodes full sequences into latent statistics, while the video pathway supplies per-frame evidence for refinement.
- Motion VAE: The motion VAE uses a bidirectional GRU with average pooling to encode motion, an MLP to produce latent mean and variance, and a forward GRU decoder.The encoder processes motion sequences and the decoder reconstructs output motion sequentially.
- Spatiotemporal feature extraction: STE is a two-layer bidirectional GRU with hidden size 1024 that outputs temporally correlated features at each timestep.The temporal feature pathway also produces a 512-dimensional latent code for the whole motion sequence.
- Motion refinement: MRR uses two fully connected layers with 1024 neurons and iteratively refines pose, shape, and camera parameters for k iterations.It takes per-frame features together with initializing body and camera parameters.