Source-linked AI summary
Deep Kinematic Pose Regression
Xingyi Zhou, Xiao Sun, Wei Zhang, Shuang Liang, Yichen Wei
TL;DR
Articulated pose estimation must handle high-dimensional motion while preserving geometric constraints. This paper embeds a differentiable kinematic model as a neural-network layer and reports state-of-the-art 3D human-pose results on Human3.6M, while leaving expressive human-body angle constraints for future work.
Problem
Articulated pose estimation is difficult because poses are high dimensional and structurally constrained, while common representations do not preserve complex articulated geometry well.
Method
The method predicts motion parameters with a deep network, converts them to joints through a differentiable kinematic layer, and trains using joint-location loss.
Results
The approach is effective on a toy problem and achieves state-of-the-art 3D human pose estimation from single-view RGB images on Human3.6M.
Takeaways & Limitations
Embedding the geometric model enables end-to-end learning while preserving articulated-object geometry and avoiding separate post-processing.
Takeaways & Limitations
Expressive human-body angle constraints remain difficult to formulate, and testing assumes an unknown subject scale is set to 1.
Abstract
from arXiv · showhide
Learning articulated object pose is inherently difficult because the pose is high dimensional but has many structural constraints. Most existing work do not model such constraints and does not guarantee the geometric validity of their pose estimation, therefore requiring a post-processing to recover the correct geometry if desired, which is cumbersome and sub-optimal. In this work, we propose to directly embed a kinematic object model into the deep neutral network learning for general articulated object pose estimation. The kinematic function is defined on the appropriately parameterized object motion variables. It is differentiable and can be used in the gradient descent based optimization in network training. The prior knowledge on the object geometric model is fully exploited and the structure is guaranteed to be valid. We show convincing experiment results on a toy example and the 3D human pose estimation problem. For the latter we achieve state-of-the-art result on Human3.6M dataset.
1 Introduction
Articulated pose estimation is difficult because poses are high-dimensional and structurally constrained. The paper embeds a differentiable kinematic model into deep learning to preserve valid geometry end to end.
- The framework predicts global position and rotation angles, maps them through a kinematic layer to joints, and trains with joint loss.
- Articulated pose estimation is especially challenging because it is high dimensional and inherently structured.
- Non-parametric and low-dimensional representations learn structure from data but cannot reliably preserve complex articulated constraints.The cited approaches include direct point prediction, PCA, sparse coding, and auto-encoders.
- A differentiable kinematic function incorporates known bone lengths, connections, and joint rotations into gradient-based network training.
- The approach extends prior depth-based hand-pose work to a toy problem and human pose estimation from RGB images.
- The method achieves state-of-the-art results for 3D human pose estimation from single-view RGB images on Human3.6M.
2 Related Work
Prior pose-estimation methods represent structural relationships using graphs, linear bases, embeddings, retrieval, explicit models, or deep networks. This paper integrates a kinematic model into deep learning and reports state-of-the-art 3D human-pose results.
- Pose Representation: Pictorial structure methods model joints and relations as graph vertices and edges but typically require complex, slow inference.
- Pose Representation: Linear dictionaries and low-dimensional embeddings encode pose structure with templates, bases, PCA, sparse coding, or auto-encoders.
- Pose Representation: Retrieval-based methods avoid explicit pose representation, but performance depends on database quality and large-scale nearest-neighbor search can be inefficient.
- Pose Representation: Explicit geometric models represent motion and shape parameters but usually estimate them with heavy optimization rather than learning.
- Deep Learning on Human Pose Estimation: Deep networks became the leading approach for human pose estimation, including heat-map supervision, hourglass architectures, and residual networks.
- Deep Learning on Human Pose Estimation: Integrating a kinematic object model into deep learning achieves state-of-the-art results for 3D human pose estimation from single-view RGB images.
3 Deep Kinematic Pose Estimation
The proposed kinematic layer embeds an articulated object's geometric model into deep pose learning, mapping motion parameters to structurally valid joint locations. Compared with direct joint or direct parameter baselines, it provides differentiable, geometry-aware learning and uses a more compact output space.
- 3.1 Kinematic Model: An articulated object is represented as bones and joints arranged in a tree, with fixed bone lengths and a root defining global position and orientation.Each joint has local rotation and translation transformations determined by the model motion parameters.
- 3.1 Kinematic Model: The forward kinematic function maps motion parameters to joint locations by multiplying transformation matrices along each joint's path from the root.The output coordinates are defined for 2D or 3D joints, with axis assignments and canonical translations specified by the model.
- 3.1 Kinematic Model: Because the kinematic function is differentiable, it can be inserted as a neural-network layer and optimized through back-propagation.The implementation differentiates the matrix corresponding to each rotational parameter while leaving the other matrix factors unchanged.
- 3.2 Deep Learning with a Kinematic Layer: The kinematic-joint method predicts motion parameters and uses a parameter-free kinematic layer to produce joints for the joint-location loss.The three compared methods share the same convolutional network and differ only in their ending parts, enabling a fair comparison.
- 3.2 Deep Learning with a Kinematic Layer: Unlike direct joint prediction, the proposed method incorporates bone lengths and spatial relations, guarantees geometrically valid joints, and uses a more compact output space.In one example, the motion-parameter space has 27 degrees of freedom versus 51 for the joint space.
- 3.2 Deep Learning with a Kinematic Layer: Direct parameter prediction is difficult because parameter impacts are uneven and joint locations may correspond ambiguously to motion angles, whereas joint-location loss is more distributed and less ambiguous.Direct joint prediction can also produce geometrically invalid outputs; ModelFit instead recovers a kinematic model through post-processing.
4 Experiment
The experiments evaluate kinematic pose regression on a synthesized 2D toy problem and single-view RGB 3D human pose estimation, including comparisons with direct joint and direct parameter regression.
- 4.1 A Toy Problem: The toy task uses synthesized binary images and tests articulated models with increasing complexity.The simplest object uses 3 motion parameters; example models use 6, 8, and 10 parameters.
- 4.1 A Toy Problem: 4.8 pixels, 4.9 pixels, and 5.1 pixels are the mean joint errors for direct parameter, kinematic joint, and direct joint, respectively.All three methods achieve low joint errors on the simple task.
- 4.1 A Toy Problem: 11.8% of the 45-pixel bone length is the direct-joint regression’s bone-length standard deviation, indicating substantial geometric constraint violation.The standard deviation is 5.3 pixels.
- 4.1 A Toy Problem: Kinematic joint maintains decent joint-location and angle accuracy as model complexity increases, while direct joint becomes less stable and direct parameter can have high joint-location error.Direct parameter can have low motion-parameter error without correspondingly low joint error, whereas it performs best on joint angle.
- 4.2 3D Human Pose Regression: The Human3.6M experiment uses 17 joints, a 27-parameter kinematic human model, and data from seven subjects performing 15 actions across four camera views.Five subjects are used for training and two for testing.
- 4.2 3D Human Pose Regression: The human model uses averaged bone lengths from seven subjects and assigns each subject a fixed global scale.The input assumes a known subject bounding box and preserves image aspect ratio through border padding.
- 4.2 3D Human Pose Regression: Kinematic joint achieves the best average accuracy among the compared methods on Human3.6M.The direct-joint baseline is already state of the art because of the powerful pretrained Residual Network, but that comparison is described as unfair due to additional ImageNet training data.
- 4.2 3D Human Pose Regression: Direct motion-parameter regression has high testing error despite decreasing training loss.The paper attributes this to ambiguous rotational targets and the difficulty of learning 3D rotational angles from RGB images.
5 Conclusions
The paper concludes that articulated-object geometric models can be integrated into convolutional neural networks through end-to-end learning. This removes the inconvenient post-processing used by previous approaches and is effective for complex 3D human pose estimation.
- The method integrates an articulated-object geometric model into a convolutional neural network.
- End-to-end learning eliminates the need for inconvenient post-processing used in previous approaches.
- Experiments show the approach is effective for complex 3D human pose estimation.
- Future work will investigate more sophisticated constraints on motion parameters.