Source-linked AI summary
Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation
Sijin Li, Weichen Zhang, Antoni B. Chan
TL;DR
3D human pose estimation must handle structured joint dependencies and ambiguous images, while existing prediction methods can rely on handcrafted constraints or output only one pose. The paper learns image and pose embeddings jointly with a maximum-margin score network whose dot product ranks candidate poses. On Human3.6M, StructNet-Avg reduces overall error by 9.2% versus DconvMP-HML and produces valid poses, while multiple design choices improve performance.
Problem
Prediction-based pose methods may incompletely capture joint dependencies, scale poorly to discretized 3D spaces, and predict only one pose when partial occlusion permits several valid poses.
Method
A deep network separately embeds image features and 3D poses into a joint space, scores them by dot product, and trains the score and embeddings with a maximum-margin criterion plus auxiliary 3D joint prediction.
Results
9.2% lower overall error than DconvMP-HML, from 133.54 to 121.31 for StructNet-Avg(500)-APF, while generating valid poses.
Takeaways & Limitations
The learned joint embedding provides an efficient discriminative score function for ranking candidate 3D poses and can represent multiple valid image-pose matches.
Takeaways & Limitations
Prediction-based alternatives are limited to one pose per image, making ambiguity from partial self-occlusion difficult to handle.
Abstract
from arXiv · showhide
This paper focuses on structured-output learning using deep neural networks for 3D human pose estimation from monocular images. Our network takes an image and 3D pose as inputs and outputs a score value, which is high when the image-pose pair matches and low otherwise. The network structure consists of a convolutional neural network for image feature extraction, followed by two sub-networks for transforming the image features and pose into a joint embedding. The score function is then the dot-product between the image and pose embeddings. The image-pose embedding and score function are jointly trained using a maximum-margin cost function. Our proposed framework can be interpreted as a special form of structured support vector machines where the joint feature space is discriminatively learned using deep neural networks. We test our framework on the Human3.6m dataset and obtain state-of-the-art results compared to other recent methods. Finally, we present visualizations of the image-pose embedding space, demonstrating the network has learned a high-level embedding of body-orientation and pose-configuration.
1. Introduction
Human pose estimation is a structured-output problem because body joints are interdependent, but prediction-based methods struggle with handcrafted constraints, 3D search spaces, and ambiguity. The paper instead learns a deep score function over image-pose pairs using maximum-margin structured learning.
- Prediction-based methods map image features to joint points or detect body parts, offering straightforward and fast evaluation.
- These methods do not explicitly capture pose structure, while manually designed pairwise constraints may incompletely represent dependencies among body joints.
- 3D estimation becomes difficult when prediction methods discretize the pose space, and single-pose outputs cannot represent multiple valid poses under partial self-occlusion.
- Score-based methods evaluate image-pose pairs jointly, assigning high scores to matches and estimating the pose that maximizes the score over the pose space.
- The proposed framework jointly learns image and pose embeddings with a maximum-margin criterion, using their dot product as an efficient score function.
2. Related work
Prior approaches use handcrafted or separately learned representations for pose detection, regression, and structured prediction, with limitations in 3D search and ambiguity. This paper learns image-pose embeddings and a dot-product score directly with deep networks rather than manually specifying structured potentials.
- 2.1. 2D pose estimation via detection with deep networks: Handcrafted features such as HoG and SIFT are critical to pictorial-structure performance but may not be optimal for pose estimation.
- 2.1. 2D pose estimation via detection with deep networks: Detection-based methods are difficult to extend from 2D to 3D because discretizing joint positions creates a large, computationally expensive pose space.
- 2.2. Pose regression via deep networks: Regression-based CNN methods directly predict joint coordinates but produce only one pose, limiting their handling of ambiguity from partial self-occlusion.
- 2.2. Pose regression via deep networks: The proposed model instead assigns similar high scores to several valid image-pose pairs, allowing it to handle pose ambiguities.
- 2.3. Structured-output prediction and feature embedding: The network embeds image features and 3D poses separately into a common space and scores them by dot product, with an auxiliary 3D joint-prediction task guiding image features.
- 2.3. Structured-output prediction and feature embedding: Unlike CRF-style structured networks, this method learns image and pose embeddings and a dot-product score rather than manually designing pairwise or higher-order output potentials.
3. Maximum-margin structured learning
The framework learns image and 3D-pose embeddings whose dot product scores image-pose compatibility, using maximum-margin structured learning with pose-dependent margins and auxiliary pose prediction.
- Embedding and score function: The network separately embeds CNN image features and 3D joint coordinates into a joint space, where their dot product scores image-pose compatibility.The image embedding uses middle- and top-convolutional features, while the pose is mapped through fully connected layers.
- Maximum-margin objective: Maximum-margin training requires the ground-truth pair to outscore alternatives by a margin that increases with pose dissimilarity.The margin-rescaling loss selects a pose that maximizes the competing score plus its distance-based margin.
- SSVM interpretation: The score is equivalent to an SSVM with element-wise image-pose joint features, while the embedding functions are discriminatively learned.The SSVM weight vector can be absorbed into the learned embedding functions.
- Maximum-margin objective: The pose-dependent margin uses mean per joint error between the ground-truth and competing 3D poses.MPJPE averages Euclidean joint errors across the body joints.
- Maximum-margin objective: A zero margin loss means the ground-truth score exceeds every alternative by the required margin, whereas positive loss indicates a violated constraint.The violated alternative is selected according to the current score function and network parameters.
- Multi-task training: An auxiliary 3D pose-prediction task encourages pose information in image features and acts as a regularizer affecting the CNN and high-level image features.Its gradients do not directly update the pose embedding network or the image embedding layer.
4. Training Algorithm
Training alternates between finding the most-violated pose and updating network parameters by back-propagation. To make pose search tractable, each mini-batch uses sampled candidates together with frequently violated poses maintained in a working set.
- Alternating optimization: For each training pair, the algorithm finds the most-violated pose using the current pose-selection network, then back-propagates through maximum-margin training.The resulting tuple of ground-truth and violated poses is called extended training data.
- Candidate-set search: Candidate poses are sampled from the pose space for each mini-batch because searching the entire pose space is computationally expensive.The candidate set contains C sampled poses.
- Candidate-set search: The candidate set is augmented with the K most frequent violated poses from a maintained working set.This working set stores poses selected as most violated during training.
- Parameter updates: Selecting the most-violated pose and back-propagating its gradient can be interpreted as max-pooling over the candidate set.
- Parameter updates: The selected pose, margin cost, and pose-prediction cost are processed in mini-batches before network parameters are updated with SGD.The update uses a line-search step size in the summarized procedure.
5. Experiments
Experiments on Human3.6M evaluate approximate inference strategies for StructNet and compare them with prior methods. Averaging high-scoring training poses improves accuracy, while APF produces valid poses with similar error and the full framework outperforms the cited baselines.
- Dataset and evaluation: Human3.6M contains around 3.6 million video frames recorded with four RGB cameras and MoCap ground-truth poses.The evaluation uses five subjects for training and validation and two subjects for testing.
- Inference strategies: StructNet approximates exhaustive pose-space search using maximum-scoring training poses, averages of the top-A poses, or APF-refined averages.The pose space is continuous, making exhaustive search computationally intractable.
- Inference strategies: About 10% lower error is obtained by StructNet-Avg(500) than StructNet-Max overall, with error stabilizing between A = 500 and A = 1000.A = 500 represents approximately 0.5% of the training poses.
- Comparison with prior methods: 9.2% lower overall error is achieved by StructNet-Avg(500)-APF than DconvMP-HML, reducing MPJPE from 133.54 to 121.31, while generating valid poses.StructNet-Avg has lower error than DconvMP-HML and LinKDE on all actions.
- Ablation studies: Removing the auxiliary pose-prediction task increases pose error, indicating that the task helps training converge to a good local optimum.The auxiliary prediction performance itself is similar to the cited CNN-based method but inferior to score-function pose estimates.
- Robustness evaluation: An 8.8% lower average error than LinKDE is achieved on the online hidden test set when training separate networks for each action category.The networks use the same network parameters across action categories.
6. Visualization of image-pose embedding
The learned image-pose embedding organizes images and poses by body orientation and leg configuration. Maximum-margin training produces a smooth, aligned embedding in which pose distances reflect MPJPE and predictions closely match ground truth.
- Embedding features: The two highest-variance embedding features correspond to the person’s orientation, separating frontal and back views in both image and pose spaces.Local regions visualize average input images and average poses associated with feature values.
- PCA structure: The first PCA component encodes viewpoint, while the second encodes leg configuration relative to the camera.Large second-component values place the left leg closer to the camera; small values place the right leg closer.
- Embedding geometry: Maximum-margin constraints induce a smooth embedding topology even without using temporal frame order.The margin is based on MPJPE, requiring pose embeddings to separate according to pose distance under the learned inner product.
- Embedding geometry: 97% of maximum-score poses for training images fall within 30 MPJPE of the ground-truth pose, indicating close image-pose alignment.The alignment is reported for the maximum-scoring candidate poses on the training images.