Source-linked AI summary
Exploiting temporal information for 3D pose estimation
Mir Rayat Imtiaz Hossain, James J. Little
TL;DR
The paper addresses ambiguous 3D human pose estimation from 2D pose sequences, where independent frame predictions can become temporally incoherent. It uses a sequence-to-sequence network with LSTM units, decoder shortcuts, and temporal smoothness training, improving the previous best Human3.6M result by approximately 12.2% and producing consistent poses under noisy detections.
Problem
3D pose estimation from 2D representations is ambiguous, and estimating poses independently for each frame leads to temporally incoherent, jittery sequences.
Method
A sequence-to-sequence network maps sequences of 2D joint locations to 3D poses using layer-normalized LSTMs, decoder shortcut connections, and temporal smoothness constraints.
Results
Approximately 12.2% improvement over the previous best result on Human3.6M yields state-of-the-art performance, with meaningful 3D poses even under extremely noisy 2D detections.
Takeaways & Limitations
Temporal information helps recover temporally consistent 3D poses across sequences, including videos where the 2D pose detector fails.
Takeaways & Limitations
For higher levels of 2D detection noise, the network performs slightly worse than Martinez et al.
Abstract
from arXiv · showhide
In this work, we address the problem of 3D human pose estimation from a sequence of 2D human poses. Although the recent success of deep networks has led many state-of-the-art methods for 3D pose estimation to train deep networks end-to-end to predict from images directly, the top-performing approaches have shown the effectiveness of dividing the task of 3D pose estimation into two steps: using a state-of-the-art 2D pose estimator to estimate the 2D pose from images and then mapping them into 3D space. They also showed that a low-dimensional representation like 2D locations of a set of joints can be discriminative enough to estimate 3D pose with high accuracy. However, estimation of 3D pose for individual frames leads to temporally incoherent estimates due to independent error in each frame causing jitter. Therefore, in this work we utilize the temporal information across a sequence of 2D joint locations to estimate a sequence of 3D poses. We designed a sequence-to-sequence network composed of layer-normalized LSTM units with shortcut connections connecting the input to the output on the decoder side and imposed temporal smoothness constraint during training. We found that the knowledge of temporal consistency improves the best reported result on Human3.6M dataset by approximately $12.2\%$ and helps our network to recover temporally consistent 3D poses over a sequence of images even when the 2D pose detector fails.
1 Introduction
3D pose estimation from 2D representations is ambiguous and ill-posed, while frame-wise predictions can be temporally incoherent. The paper addresses this with a sequence-to-sequence model that uses temporal context and smoothness constraints to predict consistent 3D poses.
- Motivation: 3D pose estimation from 2D representations is an open problem complicated by projection ambiguity, occlusion, foreshortening, and limited motion-capture data.Different 3D poses can produce the same 2D projection, while collecting 3D datasets requires expensive laboratory setups.
- Approach: The paper predicts sequences of temporally consistent 3D poses from sequences of 2D poses using a sequence-to-sequence network with decoder shortcut connections.The model uses layer-normalized LSTM units and temporal smoothness constraints during training.
- Results: 12.2% improvement over the previous best result establishes state-of-the-art performance on Human3.6M.The network also achieves the lowest error for every action class in the dataset.
- Contributions: The model uses temporal context to predict consistent 3D poses and remains effective when 2D detections are extremely noisy or meaningless.The contributions include exploiting past frames and imposing temporal consistency so prediction errors are distributed smoothly across sequences.
2 Related Work
Prior work represents 3D pose with skeletons, pose dictionaries, direct image-to-3D networks, or retrieval-based methods. Temporal approaches address the jitter caused by estimating poses independently across frames.
- Representation of 3D pose: Model-based methods represent pose with kinematic skeleton parameters, while model-free methods use 3D joint locations relative to the root node.The paper chooses the model-free representation because it is simpler and low-dimensional.
- Estimating 3D pose from 2D joints: Earlier approaches infer 3D poses from 2D joints using decision trees, Taylor-based hypotheses, or nearest-neighbor retrieval from motion-capture databases.These methods use detected 2D poses to generate or retrieve candidate 3D poses.
- Deep network based methods: Deep-network methods predict 3D pose directly from images, jointly estimate 2D and 3D poses, transfer 2D knowledge, or use volumetric heatmaps and synthetic data.Several approaches extend established 2D pose-estimation architectures to 3D prediction.
- Using temporal information: Temporal methods exploit multiple frames because independent frame predictions produce incoherent and jittery pose sequences.Prior strategies include tracking-by-detection, temporal alignment, and spatio-temporal feature extraction.
3 Our Approach
The approach maps sequences of 2D poses to temporally coherent 3D poses using an LSTM sequence-to-sequence network with decoder shortcuts and a temporal smoothness loss.
- Network Design: The encoder processes a sequence of 2D poses into a fixed-size hidden representation, while the decoder predicts the 3D pose at each subsequent time step.The decoder is initialized from the encoder’s final state and receives a start token before generating the pose sequence.
- Network Design: Decoder shortcut connections make each unit predict the perturbation from the previous 3D pose rather than the full pose directly.This residual mapping is described as easier for the network to learn.
- Loss function: Temporal smoothness assumes limited subject movement and addresses frame-to-frame variation in noisy 2D detections, especially for fast-moving or occluded joints.The constraint is designed to keep each joint’s prediction from differing too much from the previous frame.
- Loss function: The temporal derivative penalty uses separate torso-head, limb-leg, and limb-arm joint sets, with scalar weights controlling their contributions.Higher weights are assigned to joint groups that are generally predicted with higher error.
- Loss function: The network combines 3D pose mean squared error with the mean L2 norm of first-order temporal derivatives across sequences.The smoothness term penalizes excessive frame-to-frame changes in predicted joint locations.
- Loss function: The overall loss weights the pose-error and temporal-smoothness terms using scalar hyper-parameters α and β.These parameters regulate the relative importance of the two loss components.
4 Experimental Evaluation
The evaluation uses Human3.6M and HumanEva under established protocols, with additional tests on ground-truth, noisy, and detector-estimated 2D poses. Across these settings, the sequence model achieves strong accuracy and robustness, while higher input noise remains a boundary.
- Datasets and protocols: The study evaluates on Human3.6M and HumanEva, using standard Human3.6M protocols and qualitative videos from Human3.6M and YouTube.Human3.6M training uses subjects 1, 5, 6, 7, and 8, while subjects 9 and 11 are used for testing.
- Human3.6M results: 12.1% improvement over Sun et al. is reported under Human3.6M protocol #1, with the lowest error for every action class.The improvement is reported as 7.2 mm, and one model is trained for all action classes.
- Human3.6M results: 8.09% improvement over reported state-of-the-art results is reported under protocol #2, again with the lowest error for each action.The improvement is reported as 3.7 mm after similarity-transform alignment.
- HumanEva results: On HumanEva, the model obtains the lowest error in four of six cases and the lowest average error for Walking and Jogging.It also obtains the second-best result for subject 2 on Walking.
- Ground-truth 2D evaluation: 13.8% improvement over Martinez et al.’s lower-bound error is reported when the model is trained with ground-truth 2D poses.With noise-free 2D joints, the network also outperforms the models by Martinez et al. and Moreno-Nouguer under protocol #2.
- Noise and detector robustness: The model is more robust than Moreno-Nouguer’s under noisy inputs, but performs slightly worse than Martinez et al. when noise standard deviation exceeds 10.The system is also evaluated on stacked-hourglass detections and achieves state-of-the-art results with a non-fine-tuned detector.
5 Conclusion
Exploiting temporal context produces accurate, temporally smooth 3D pose estimates, including on occlusion-heavy actions and noisy 2D detections. The network also supports efficient, near-real-time inference.
- The network achieved the best reported accuracy on all 15 Human3.6M action classes.
- Temporal context yields temporally smooth 3D poses and significantly improves performance on actions with substantial occlusion.
- The model is reasonably robust to noisy 2D poses, with temporal smoothness especially visible on challenging YouTube videos.
- Inference takes 3ms per frame on average, suggesting real-time application when the 2D pose detector is also real time.