Structured Prediction of 3D Human Pose with Deep Neural Networks

• Monocular images make 3D pose estimation difficult because occlusions, complex backgrounds, and lost depth information create reconstruction ambiguities.Source: §1 · p. 1
• Max-margin structured methods model joint dependencies effectively but require solving a computationally expensive optimization problem at test time.Source: §1 · p. 1
• The proposed model pretrains an overcomplete auto-encoder, maps CNN image features into its high-dimensional latent pose representation, and decodes them into pose space.Source: §1 · p. 2, Figure 1 · p. 2
• The latent representation encodes dependencies between body parts, implicitly enforces pose constraints, preserves human body statistics, and supports end-to-end fine-tuning.Source: §1 · p. 2
• Combining CNNs for supervised learning with auto-encoders for structured learning preserves CNN power while accounting for dependencies and increasing performance.Source: §1 · p. 2

• Deep learning methods for 3D pose estimation typically regress continuous 3D joint locations because discretizing 3D space is more difficult than discretizing 2D space.Source: §2 · p. 3
• Projection from 3D to 2D introduces ambiguities that motivate modeling dependencies between body joints within deep learning approaches.Source: §2 · p. 3
• An earlier auto-encoder method learned a shared representation for silhouettes and 3D poses but required accurate foreground masks and handcrafted features.Source: §2 · p. 3
• The strongest prior deep structured method extended structured SVMs by learning similarity scores between image and pose embeddings, but required costly test-time pose search.Source: §2 · p. 3
• Denoising and contractive auto-encoders had shown useful generalization or robustness, but prior work used auto-encoders primarily for feature learning.Source: §2 · p. 3

3 Method

• 3D human pose is represented as J joint locations relative to a root joint, rather than joint angles and limb lengths.The pose vector is y ∈ R3J.Source: §3 · p. 3
• An overcomplete denoising auto-encoder learns a high-dimensional latent pose representation that encodes dependencies between human joints.Its middle hidden layer has higher dimension than the input and output and is denoted hL.Source: §3 · p. 4, §3.1 · pp. 4–5
• The auto-encoder is trained on noisy pose vectors to reconstruct the original denoised poses using reconstruction loss and Jacobian regularization.The Jacobian penalty is used to improve robustness to small pose changes.Source: §3.1 · pp. 4–5
• A CNN maps input images to the auto-encoder’s learned latent pose representation using a squared loss over training samples.The CNN uses a standard convolutional architecture and linearly maps its image representation to the latent pose embedding.Source: §3.2 · p. 5
• The auto-encoder’s decoding layers are appended to the CNN to reproject latent estimates into pose space, after which the complete network is fine-tuned.Fine-tuning minimizes squared differences between predicted and ground-truth 3D poses.Source: Figure 2 · p. 4, §3.3 · p. 6

Evaluation protocol

• The dataset contains 3.6 million image frames with corresponding 2D and 3D poses across complex activities and 4 camera viewpoints.Images were cropped around subjects and resized to 128×128, with outputs represented by 17 3D joint coordinates.Source: §4.1 · p. 6
• Human3.6m evaluation used 5 subjects for training, 2 different subjects for testing, and average Euclidean joint distance in millimeters.Training and testing were performed monocularly across all camera views for each separate action.Source: §4.3 · pp. 6–7, §4.1 · p. 6

Comparison with prior methods

• Our method consistently outperforms KDE regression, joint detector-regressor networks, and max-margin structured prediction on Human3.6m.The max-margin baseline used either the highest-scoring pose or an average of the 500 highest-scoring training poses.Source: §4.4 · p. 7

Ablation studies

• Both one- and two-layer auto-encoders perform similarly and better than CNN-Direct, especially for Taking Photo and Walking Dog.These actions involve environmental interactions and are therefore more physically constrained.Source: §4.4 · pp. 7–8
• Fine-tuning consistently improves the results, while the overcomplete auto-encoder outperforms added fully connected layers and PCA-based latent representations.For PCA, the 40-dimensional configuration performed best, but the 40-dimensional auto-encoder performed better.Source: §4.4 · p. 8

Structure preservation

• Predictions from the proposed method better preserve ground-truth limb ratios than KDE and CNN-Direct, indicating better modeling of joint dependencies.The comparison uses differences between ground-truth and predicted limb ratios.Source: §4.4 · p. 8

Conclusion

• The approach combines auto-encoders with CNNs to account for dependencies between human body parts efficiently while improving prediction accuracy over state-of-the-art methods.The framework is presented as generic and intended for structured prediction beyond human pose.Source: §5 · p. 8
• Our method outperforms other methods on upper-body and full-body limb-ratio errors, although all methods perform well on the lower body.Figure 4 compares KDE, CNN-Direct, and the proposed method with and without fine-tuning.Source: Figure 4 · p. 9