Source-linked AI summary
HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation
Kun Zhou, Xiaoguang Han, Nianjuan Jiang, Kui Jia, Jiangbo Lu
TL;DR
Single-image 3D pose estimation is ambiguous, and high-quality 3D labels for in-the-wild images are difficult to obtain. HEMlets encode part-centric relative depth between joints as an intermediate representation before volumetric joint-heatmap regression. The method reports 39.9 mm MPJPE on Human3.6M, about 20% better than the best-of-grade method.
Problem
Single-image 3D pose estimation is ambiguous, and obtaining high-quality 3D annotations for in-the-wild images requires substantial effort.
Source: §1 · p. 1Method
HEMlets represent part-centric relative depth and joint co-location between connected joints, providing an intermediate state for 3D pose learning.
Source: §1 · pp. 1–2, §3.1 · p. 3Results
Takeaways & Limitations
Training can use relative-depth annotations as well as 2D or 3D joint annotations.
Source: §3.2 · p. 5
Abstract
from arXiv · showhide
Estimating 3D human pose from a single image is a challenging task. This work attempts to address the uncertainty of lifting the detected 2D joints to the 3D space by introducing an intermediate state - Part-Centric Heatmap Triplets (HEMlets), which shortens the gap between the 2D observation and the 3D interpretation. The HEMlets utilize three joint-heatmaps to represent the relative depth information of the end-joints for each skeletal body part. In our approach, a Convolutional Network (ConvNet) is first trained to predict HEMlests from the input image, followed by a volumetric joint-heatmap regression. We leverage on the integral operation to extract the joint locations from the volumetric heatmaps, guaranteeing end-to-end learning. Despite the simplicity of the network design, the quantitative comparisons show a significant performance improvement over the best-of-grade method (by 20% on Human3.6M). The proposed method naturally supports training with "in-the-wild" images, where only weakly-annotated relative depth information of skeletal joints is available. This further improves the generalization ability of our model, as validated by qualitative comparisons on outdoor images.
1. Introduction
Single-image 3D pose estimation is ambiguous, and high-quality 3D annotations for in-the-wild images are costly to obtain. HEMlets add part-centric relative-depth cues to 2D joint information, supporting a staged learning approach that achieves a strong Human3.6M result.
1. Introduction
- Single-image 3D pose inference is ambiguous, while obtaining high-quality 3D labels for in-the-wild images requires substantial effort.The paper identifies representation efficiency and learning effectiveness as another challenge for regression-based approaches.Source: §1 · p. 1
- HEMlets encode relative depth for kinematically connected joint pairs, with dense per-pixel co-location likelihoods and less exposure to inconsistent annotations.The representation uses polarized heatmaps to capture local depth relationships.Source: §1 · pp. 1–2
- 39.9 mm MPJPE on Human3.6M, about 20% better than the best-of-grade method.The paper also reports state-of-the-art results across standard benchmarks.Source: §1 · p. 2
- Progressive learning decomposes 3D pose estimation into easier subtasks, using 2D joint detection and HEMlets as intermediate supervision.The paper describes HEMlets as easier to train and less prone to overfitting, and notes they can use labeled relative-depth datasets.Source: §1 · p. 2
- HEMlets extend 2D per-joint heatmaps with additional heatmaps for dense local depth ordering while retaining soft-argmax end-to-end learning.This representation is intended to balance representational power with learning effectiveness.Source: §1 · p. 2
2. Related Work
Prior work moves from direct image-to-pose prediction and 2D-to-3D lifting toward 3D-aware intermediate supervision. HEMlets are presented as a part-centric representation that can also use weak relative-depth labels for in-the-wild adaptation.
2.1. Direct encoder-decoder
- Direct encoder-decoder methods infer pose from images, but their reliance on 3D pose datasets limits their use of large-scale 2D pose data.Examples include multitask CNNs, an autoencoder for joint dependencies, and fusion of 2D locations with image features.Source: §2.1 · p. 2
2.2. Transition with 2D joints
- Two-stage methods first estimate 2D joints and then lift them to 3D, but the 2D projection leaves an ambiguity gap to the 3D pose.The cited approaches include residual regression, pose grammar, adversarial learning, and reprojection-based methods.Source: §2.2 · p. 2
2.3. 3D-aware intermediate states
- 3D-aware intermediate states bridge image features and 3D joints; HEMlets encode 2D locations and relative depth for part-centric joint pairs.Prior approaches include volumetric joint heatmaps and relative-depth supervision [25]; HEMlets are presented as balancing representation efficiency and learning effectiveness.Source: §2.3 · pp. 2–3
2.4. “In-the-wild” adaptation
- Weak relative-depth labels are easier to obtain than exact 3D coordinates and can support training or adaptation on in-the-wild images.Prior work uses relative-depth annotation to make larger in-the-wild datasets feasible [25].Source: §2.4 · p. 3
3. HEMlets Pose Estimation
HEMlets represent local skeletal-part joint locations and relative depth ordering together, allowing images with relative-depth annotations to be used alongside images with full 3D joint annotations. The network predicts HEMlets and 2D joint heatmaps, then regresses volumetric joint heatmaps to obtain 3D coordinates.
3.1. Part-centric heatmap triplets
- For 18 joints partitioned into 14 parts, each parent–child pair is encoded by three polarized heatmaps, co-locating the joints when their depths are similar and otherwise separating the child by polarity.The depth threshold ε adjusts sensitivity to the relative depth difference.Source: §3.1 · pp. 3–4
- HEMlets jointly represent pairwise 2D co-location likelihoods and ordinal depth relations, implicitly learning geometric constraints such as bone lengths.Unlike Pavlakos et al., HEMlets models these signals jointly rather than independently marginalizing the 2D locations and depth.Source: §3.1 · p. 4
3.2. 3D pose inference
- A ResNet-50 and two upsampling branches predict HEMlets and 2D heatmaps, combine them into a latent representation, and join it with image features for volumetric joint prediction.Soft-argmax aggregates the 3D feature maps into joint estimates.Source: §3.2 · p. 4
- The training objective combines masked L2 losses for HEMlets and 2D heatmaps with an L1 loss for regressed 3D joints.The HEMlets loss masks out missing annotations.Source: §3.2 · pp. 4–5
- Joint HEMlets and 2D-heatmap supervision allows training with images annotated in 2D or with exact 3D coordinates or relative depth ordering.The paper treats exact 3D joint coordinates and relative-depth ordering between part-centric joint pairs as 3D annotations.Source: §3.2 · p. 5
4. Experiments
The experiments evaluate the method on three benchmarks, compare it with prior methods, and test design choices through ablations. They also report qualitative results on in-the-wild images and examine training variants and data augmentation.
4.1. Datasets and evaluation protocols
- Human3.6M uses Protocol #1 with five training subjects and two evaluation subjects; HumanEva-I uses its training sequences, while MPI-INF-3DHP is tested without its training set.The paper reports MPJPE for Human3.6M and HumanEva-I, and 3DPCK and AUC for MPI-INF-3DHP.Source: §4.1 · p. 5
4.2. Results and comparisons
- 20% higher 3D pose prediction accuracy than Sun et al. under Protocol #1; the method also outperforms competitors across action subjects and protocols.The comparison uses Human3.6M; Table 1 reports MPJPE under Protocols #1 and #2 and PA MPJPE.Source: Table 1 · p. 6, §4.2 · p. 6
- On HumanEva-I, the method outperforms prior approaches; on MPI-INF-3DHP, it trails by one point on Studio GS but leads on Outdoor and Studio no GS.For MPI-INF-3DHP, the evaluated model was trained on Human3.6M without training data from that dataset.Source: §4.2 · pp. 6–7, Table 3 · p. 7
4.3. Ablation study
- With MPII data, 2D-heatmap supervision lowers error by 3.0mm versus baseline, HEMlets lowers it by 1.7mm versus 2D supervision, and the combined model achieves the lowest error.Without MPII, HEMlets without L2D reaches 46.0mm and the full model 45.1mm; Table 4 distinguishes this Human3.6M-only setting.Source: §4.3 · p. 7, Table 4 · p. 7
- In the pictured comparison, similar 2D errors accompany 3D errors of 81.3mm and 41.2mm; HEMlets learning is reported to fix gross limb errors.The full model takes 13.3ms per inference with 47.7M parameters, versus 8.5ms and 34.3M for the baseline on a GTX 1080.Source: §4.3 · p. 7
- The five-state and two-state HEM variants converge worse than HEMlets when trained on Human3.6M under the same experiment setting.Figure 6 compares their validation losses.Source: §4.3 · p. 7, Figure 6 · p. 8
- Adding Ordinal or FBI annotations slightly increases 3DPCK on MPI-INF-3DHP, with FBI annotations scoring better than Ordinal annotations.The authors suggest differences in manual annotation errors across annotation schemes may explain the result.Source: §4.3 · p. 7
- Qualitative predictions on 3DHP and LSP include visually correct results for challenging poses such as self-occlusion and upside-down poses.The examples include the validation set of MPI-INF-3DHP and in-the-wild Leeds Sports Pose images.Source: §4.3 · p. 8, Figure 5 · p. 8
5. Conclusion
The conclusion presents HEMlets as an intermediate representation for single-image 3D pose estimation and reports improved benchmark accuracy and generalization. It also suggests the representation may benefit other 3D regression problems.
5. Conclusion
- HEMlets encode each skeletal part’s joints’ relative forward-or-backward depth relation and spatial co-location likelihoods, bridging the input image and output 3D pose.The conclusion describes HEMlets as easy to learn.Source: §5 · p. 8
- About 20% relative accuracy improvement over the best-of-grade method is reported on Human3.6M.The conclusion also reports good generalization ability.Source: §5 · p. 8
- The authors suggest HEMlets may benefit other 3D regression problems, such as scene depth estimation.This is presented as a potential application.Source: §5 · p. 8