Source-linked AI summary
VIBE: Video Inference for Human Body Pose and Shape Estimation
Muhammed Kocabas, Nikos Athanasiou, Michael J. Black
TL;DR
Video pose and shape estimation lacks reliable in-the-wild 3D motion supervision, limiting accurate and natural sequence prediction. VIBE combines a temporal regressor with an AMASS-based motion discriminator, achieving strong benchmark performance and showing the value of modeling motion.
Problem
Existing video methods lack reliable in-the-wild ground-truth 3D annotations, while prior temporal models insufficiently capture the complexity and variability of real human motion.
Method
VIBE uses a temporal pose-and-shape regressor with adversarial sequence training, where a motion discriminator compares predicted sequences against AMASS motion-capture sequences.
Results
VIBE outperforms state-of-the-art frame-based and temporal methods on challenging 3DPW and MPI-INF-3DHP benchmarks, while improving over single-frame methods by a significant margin.
Takeaways & Limitations
Sequence-level adversarial training with AMASS produces realistic motion and demonstrates that exploiting human motion improves video-based 3D pose and shape estimation.
Takeaways & Limitations
Future work should investigate dense motion cues, multi-person disambiguation, occlusion tracking, and alternative attentional encodings such as transformers.
Abstract
from arXiv · showhide
Human motion is fundamental to understanding behavior. Despite progress on single-image 3D pose and shape estimation, existing video-based state-of-the-art methods fail to produce accurate and natural motion sequences due to a lack of ground-truth 3D motion data for training. To address this problem, we propose Video Inference for Body Pose and Shape Estimation (VIBE), which makes use of an existing large-scale motion capture dataset (AMASS) together with unpaired, in-the-wild, 2D keypoint annotations. Our key novelty is an adversarial learning framework that leverages AMASS to discriminate between real human motions and those produced by our temporal pose and shape regression networks. We define a temporal network architecture and show that adversarial training, at the sequence level, produces kinematically plausible motion sequences without in-the-wild ground-truth 3D labels. We perform extensive experimentation to analyze the importance of motion and demonstrate the effectiveness of VIBE on challenging 3D pose estimation datasets, achieving state-of-the-art performance. Code and pretrained models are available at https://github.com/mkocabas/VIBE.
1. Introduction
VIBE addresses inaccurate video-based 3D pose and shape estimation by combining temporal modeling with adversarial training on AMASS motion data and unpaired in-the-wild 2D annotations. It produces plausible motion sequences and outperforms prior methods on challenging benchmarks.
- Problem: Video methods struggle because in-the-wild ground-truth 3D annotations are scarce, while existing datasets and pseudo-labels inadequately cover subjects, motions, image complexity, and reliable 2D supervision.These limitations constrain training for accurate and natural motion estimation.
- Method: VIBE trains a temporal model to predict SMPL pose and shape parameters while a motion discriminator distinguishes regressed sequences from AMASS motions.The discriminator provides weak supervision through an adversarial loss.
- Method: The motion discriminator implicitly models static, physical, and kinematic properties of human motion using ground-truth motion-capture data.This encourages the regressor to produce valid human motions without in-the-wild 3D labels.
- Method: VIBE combines a pretrained single-image CNN with a temporal encoder, body-parameter regressor, and GRU-based motion discriminator for sequence prediction.During training, predicted poses are compared with AMASS samples at the sequence level.
- Results: VIBE outperforms prior frame-based and temporal methods on 3DPW and MPI-INF-3DHP, and consistently beats single-frame methods by a significant margin.The paper also reports state-of-the-art performance on major 3D pose-estimation benchmarks.
2. Related Work
Prior work spans static body-model estimation, video-based motion recovery, and adversarial sequence modeling. VIBE combines these directions around temporal SMPL estimation and motion discrimination using large-scale human-motion data.
- 3D pose and shape from a single image: Parametric 3D human body models such as SMPL provide pose and shape outputs, capture body-shape statistics, and yield meshes for downstream tasks.Earlier approaches include bottom-up regression, top-down optimization, and multi-camera methods.
- VIBE architecture: VIBE estimates SMPL body-model parameters per video frame with a temporal generation network trained alongside a motion discriminator.The discriminator accesses a large corpus of human motions represented in SMPL format.
- 3D pose and shape from video: Video pose methods include motion-prior approaches, 2D-to-3D lifting, end-to-end 3D regression, and temporal extensions of SMPLify.Several methods perform well indoors but generalize poorly to in-the-wild datasets such as 3DPW and MPI-INF-3DHP.
- GANs for sequence modeling: Sequential adversarial models have been used for sequence-to-sequence tasks and for predicting or generating human-motion sequences.This line of work motivates applying GAN-based sequence modeling to video pose estimation.
3. Approach
VIBE uses a temporal encoder to estimate SMPL pose and shape parameters from video, while adversarial motion modeling encourages realistic sequences. Its discriminator compares generated motions with AMASS sequences, using recurrent processing and self-attention to judge temporal plausibility.
- Temporal Encoder: VIBE extracts frame features, encodes temporal information with bidirectional GRUs, and regresses SMPL pose and shape parameters for each frame.The temporal encoder incorporates information from past and future frames before body-parameter regression.
- Temporal Encoder: The SMPL representation contains 72 pose parameters and 10 shape parameters, producing a posed 3D mesh from these values.Pose includes global and joint rotations, while shape uses the first 10 coefficients of a PCA shape space.
- Temporal Encoder: For each video, the generator predicts time-varying poses and averages framewise shape estimates into one sequence-level shape prediction.The resulting pose-and-shape sequence is passed to the motion discriminator alongside AMASS samples.
- Temporal Encoder: The generator combines available 2D, 3D, pose, shape, and adversarial losses to train temporally consistent body estimates.The 2D loss projects SMPL joints through a weak-perspective camera model with scale, translation, rotation, and orthographic projection.
- Motion Discriminator: The motion discriminator distinguishes generated pose sequences from real AMASS motions, learning plausible configurations from ground-truth poses.This sequence-level constraint addresses cases where individually plausible poses form an unrealistic movement sequence and can reduce reliance on a separate single-frame discriminator.
- Motion Discriminator: The discriminator uses GRU layers followed by self-attention, which emphasizes important frames for the real/fake sequence decision.The paper compares this dynamic weighting with average-plus-max static pooling and also evaluates MPoser, a sequential VAE motion prior used as a regularizer instead of the discriminator.
- Implementation: The implementation uses a frozen pretrained ResNet-50 frame encoder and selects a sequence length of T = 16 after testing lengths from 8 to 128.The selected configuration was chosen because it yielded the best results and fit training on a single RTX2080ti GPU.
4. Experiments
Experiments evaluate VIBE against state-of-the-art methods across major 3D pose benchmarks and test the contributions of motion discrimination and self-attention. VIBE improves challenging in-the-wild performance, while ablations show consistent benefits from the motion discriminator and dynamic temporal aggregation.
- Comparison to state-of-the-art results: VIBE outperforms previous frame-based and temporal methods on challenging in-the-wild 3DPW and MPI-INF-3DHP datasets, while matching SPIN on Human3.6M.The model trained with additional 3DPW data improves in-the-wild performance; the direct-comparison model uses datasets similar to Temporal-HMR.
- Comparison to state-of-the-art results: VIBE improves MPJPE and PVE through greater temporal pose and shape consistency, although Temporal-HMR produces smoother predictions through aggressive smoothing.Temporal-HMR’s smoothing sacrifices accuracy on videos with fast motion or extreme poses, revealing a trade-off between accuracy and smoothness.
- Ablation Experiments: Adding the motion discriminator DM improves the temporal generator across baselines while producing smoother predictions.The improvement holds with both HMR and SPIN pretrained feature extractors, and Table 2 reports consistent gains over all baselines.
- Ablation Experiments: 4.5mm PVE for MPoser on held-out AMASS data versus 6.0mm for frame-based VPoser shows that modeling motion improves pose-sequence regularization.MPoser remains weaker than DM for in-the-wild prediction, partly because AMASS does not cover every motion found in such videos.
- Ablation Experiments: Dynamic feature aggregation with self-attention significantly outperforms static pooling by weighting individual frames according to their temporal relevance.The attention mechanism learns frame correlations and produces a more fine-grained representation of the motion sequence.
5. Conclusion
VIBE combines recurrent temporal modeling, adversarial motion discrimination, self-attention, and an AMASS-based motion prior to improve video pose and shape estimation. Experiments and qualitative comparisons support the value of video training while identifying future directions for richer motion cues and tracking.
- Contributions: VIBE introduces recurrent temporal modeling and adversarial discrimination of motion sequences using AMASS.The method also incorporates self-attention in the discriminator and evaluates an AMASS-learned motion prior.
- Contributions: Ablation studies show how the proposed design choices contribute to state-of-the-art performance on video benchmark datasets.The paper presents these evaluations as evidence for the value of training from video.
- Future work: Future work includes optical-flow cues, multi-person disambiguation, occlusion tracking, and transformer-based attention for body-kinematics estimation.The paper also proposes using video to supervise single-frame methods by fine-tuning HMR features.
- Qualitative results: Qualitative results compare VIBE with Temporal-HMR on challenging sequences containing fast motion, extreme poses, and self occlusion.The comparison reports more accurate poses for VIBE than Temporal-HMR.
6. Appendix – Supplmentary Material
The supplementary material specifies VIBE’s pose generator, motion discriminator, loss weights, datasets, evaluation setup, and metrics. These details cover the recurrent architecture, training components, benchmark data, and procedures used for comparison.
- Pose Generator: The pose generator extracts ResNet50 features, processes them with a two-layer GRU, and regresses SMPL pose and shape parameters.A residual connection assists training, and the SMPL regressor is initialized from pretrained HMR weights.
- Motion Discriminator: The motion discriminator uses two GRU layers with hidden size 1024 and, in the best model, self-attention with two 1024-neuron MLP layers.The self-attention configuration uses dropout rate 0.1.
- Loss: The loss uses λ2D=300, λ3D=300, λβ=0.06, λθ=60, and λLadv=2 for its keypoint, parameter, and adversarial terms.The implementation also uses two GRU layers with hidden dimension 1024.
- Datasets: Training and evaluation use datasets spanning controlled indoor, in-the-wild, multi-action, and multi-person video settings.The listed datasets include MPI-INF-3DHP, Human3.6M, 3DPW, PennAction, InstaVariety, and PoseTrack.
- Evaluation: Evaluation follows the setup of prior work and reports MPJPE, PA-MPJPE, PVE, PCK, and acceleration error.MPJPE measures pelvis-centered mean joint-position error, while the other metrics assess aligned pose, mesh vertices, keypoint correctness, and acceleration.