Source-linked AI summary
Compositional Human Pose Regression
Xiao Sun, Jiaxiang Shang, Shuang Liang, Yichen Wei
TL;DR
Regression methods often underuse pose structure by estimating joints independently. This paper introduces compositional pose regression using bones and structure-aware losses, achieving state-of-the-art results on Human3.6M and competitive regression performance on MPII.
Problem
Previous regression methods independently minimize per-joint errors and therefore do not adequately exploit joint dependence or pose structure.
Method
Compositional pose regression represents poses with bones and uses joint connection structure to define a loss encoding long-range interactions between bones.
Results
The approach advances the state of the art on Human3.6M and is competitive with detection-based methods on MPII, reaching 59.1 mm average joint error on Human3.6M.
Takeaways & Limitations
Structure-aware regression is competitive for 2D pose estimation and has greater potential for 3D pose estimation, where complex structural constraints are critical.
Takeaways & Limitations
Prior evaluations focused only on joint location accuracy, limiting assessment of internal pose structure.
Abstract
from arXiv · showhide
Regression based methods are not performing as well as detection based methods for human pose estimation. A central problem is that the structural information in the pose is not well exploited in the previous regression methods. In this work, we propose a structure-aware regression approach. It adopts a reparameterized pose representation using bones instead of joints. It exploits the joint connection structure to define a compositional loss function that encodes the long range interactions in the pose. It is simple, effective, and general for both 2D and 3D pose estimation in a unified setting. Comprehensive evaluation validates the effectiveness of our approach. It significantly advances the state-of-the-art on Human3.6M and is competitive with state-of-the-art results on MPII.
1. Introduction
Regression methods are general for 2D and 3D pose estimation but underperform detection-based methods because they do not exploit pose structure. Compositional pose regression addresses this by representing poses with bones and using a structure-aware loss, while remaining broadly compatible and effective.
- Regression methods directly predict joints and support both 2D and 3D estimation, but generally underperform detection-based methods.Detection-based methods infer joints from heat maps, whereas regression methods directly map images to joint coordinates.
- Independent per-joint regression ignores joint dependence and internal pose structure, leaving geometric constraints such as fixed bone lengths unsatisfied.
- Compositional pose regression represents poses with bones and defines a loss that uses joint connections to encode long-range interactions between bones.The representation is motivated by bones being more primitive, stable, and easier to learn than joints.
- The approach changes the pose representation and loss while preserving other algorithm choices, allowing adaptation to existing regression methods with little memory or computation overhead.
- The unified approach handles 2D and 3D regression and can mix both data types during training.
- 59.1 mm average joint error on Human3.6M was reported as about 12% relatively better than the state of the art, while MPII reached 86.4% PCKh 0.5.
2. Related Work
Prior work exploits pose structure through graphical models, multi-stage networks, latent representations, or kinematic models, while approaches using in-the-wild 2D data often separate 2D and 3D learning. The proposed approach instead combines structural modeling with unified 2D–3D training.
- 3D Pose Estimation: Earlier 3D methods either reconstruct pose from predicted 2D joints without end-to-end learning or implicitly learn dependencies through latent representations and coarse-to-fine prediction.
- 3D Pose Estimation: Kinematic-model approaches preserve geometric structure but use highly nonlinear parameterizations, are difficult to optimize, and do not generalize to 2D.
- 2D Pose Estimation: Traditional 2D graphical-model methods formulate pose estimation as graph inference, but inference is typically complex, slow, and difficult to integrate with deep networks.
- 2D Pose Estimation: Deep 2D pose methods integrate graphical models into CNNs or use multi-stage architectures to learn spatial relationships and refine predictions.
- Leveraging in the wild 2D data for 3D pose estimation: Human3.6M is limited by restricted subject, environmental, and pose variation, which reduces generalization to in-the-wild images.
- Leveraging in the wild 2D data for 3D pose estimation: Methods leveraging in-the-wild 2D data for 3D estimation commonly use separate stages, pretraining and fine-tuning, or separate networks for 2D and 3D tasks.
- Leveraging in the wild 2D data for 3D pose estimation: The proposed approach treats 2D and 3D data uniformly in one training framework, fully exploiting abundant 2D information while remaining compatible with existing techniques.
- Leveraging in the wild 2D data for 3D pose estimation: Synthetic-data methods are complementary to these approaches because they focus on data augmentation.
3. Compositional Pose Regression
The method replaces independent joint regression with bone-based pose representation and a compositional loss that exploits long-range skeleton structure. This addresses unstable bone prediction and structural errors while retaining efficient regression.
- Motivation: Direct joint regression independently estimates joints, ignoring joint dependence and potentially violating geometric constraints such as fixed bone lengths.
- Bone-Based Representation: Bones are represented as directed local vectors along the pose tree, with the pelvis as root and the origin as the root’s parent.
- Bone-Based Representation: Bone standard deviations are much smaller than corresponding joint standard deviations, especially for distal parts, in both Human3.6M and MPII.
- Bone-Based Representation: Bone-based representation expresses geometric constraints more directly because fixed-length and joint-angle constraints involve fewer bone variables than joint variables.
- Compositional Loss: Bone-only regression can accumulate individual prediction errors along the skeleton, producing large and random errors for distant joints such as the wrist.
- Compositional Loss: The compositional loss sums unnormalized bones along paths between joint pairs, jointly balancing long-range errors across intermediate bones.
- Compositional Loss: The joint-pair set can range from global joint pairs or adjacent bone pairs to all pairs, with the all-pairs variant fully exploiting pose structure.
4. Unified 2D and 3D Pose Regression
The same pose-regression formulation supports both 2D and 3D outputs and can train on mixed-dimensional data. Coordinate components and losses are separated so depth contributes only for 3D samples.
- Unified Formulation: All equations apply to both 2D and 3D pose estimation, with network outputs containing either 2K or 3K coordinates.
- Mixed-Dimensional Training: Mixed 2D and 3D training decomposes joints, bones, and relative positions into xy and z components, with corresponding decomposed losses.
- Mixed-Dimensional Training: The xy loss is computed for both modalities, whereas the z loss is computed only for 3D samples and contributes no gradient for 2D samples.
- Coordinate Handling: During inference, 3D metric coordinates are recovered by backprojecting the xy component using known camera intrinsics after appropriate normalization handling.
- Training: Training uses the same ResNet-50-based setup across 3D, 2D, and mixed tasks, with ImageNet pretraining and SGD optimization.
- Training: Input images are resized to 224 × 224 and augmented through translation, scaling, rotation, and flipping; mixed batches contain equal numbers of 2D and 3D samples.
5. Experiments
Experiments evaluate the approach on Human3.6M and MPII using standard and structure-focused metrics, ablations, and comparisons with prior work. Results show gains from 2D data, bone representation, and compositional loss, alongside strong benchmark performance.
- Datasets: Human3.6M and MPII provide the 3D and 2D evaluation benchmarks, respectively, with different data characteristics and pose annotations.Human3.6M is captured in a controlled environment with motion-capture annotations, whereas MPII contains complex images and annotated 2D poses from daily activities.
- Evaluation metrics: Bone Error, Bone Std, and Illegal Angle complement joint-location metrics by measuring relative accuracy, bone-length stability, and physically feasible limb-joint angles.Bone Error applies to both 2D and 3D pose, while Bone Std and Illegal Angle evaluate 3D geometric validity.
- Ablation study: Reducing joint error from 102.2 to 64.2 shows that adding MPII data improves all reported metrics in the Human3.6M experiments.The authors attribute this improvement to better learned features from abundant 2D data and treat the addition of MPII as a baseline rather than a main contribution.
- Ablation study: Bone-based variants outperform comparable joint-based variants on nearly all metrics, especially those measuring geometric constraints.The comparison uses roughly the same amount of supervision for the joint- and bone-based variants.
- Ablation study: Using the compositional loss, Ours (all) improves the Human3.6M-only baseline by 9.8 mm on Joint Error and 7.5 mm on PA Joint Error.It also improves Bone Error by 7.1 mm, Bone Std by 4.7 mm, and Illegal Angle by 1.2%.
- Comparison with prior work: Ours (all) improves the state of the art by 7 mm, 7.8 mm, and 14.9 mm across the three Human3.6M comparison settings.The corresponding relative improvements are 12.7%, 11.7%, and 13.9%; comparisons are not completely fair because training data, architectures, and implementations differ.
6. Conclusion
The paper shows that exploiting pose structure makes regression competitive with leading detection-based methods for 2D estimation, while offering greater potential for 3D estimation.
- Structure-aware regression is competitive with leading detection-based approaches for 2D pose estimation.
- The approach has greater potential for 3D pose estimation, where more complex structural constraints are critical.
- The central conclusion is that regression performance improves when pose structure is appropriately exploited.