Source-linked AI summary
Ordinal Depth Supervision for 3D Human Pose Estimation
Georgios Pavlakos, Xiaowei Zhou, Kostas Daniilidis
TL;DR
The paper tackles the limited availability of accurate 3D annotations for natural images in single-image human-pose estimation. It trains ConvNets using ordinal joint-depth relations, extends LSP and MPII with these annotations, and reports competitive performance and state-of-the-art benchmark results. The approach is intended for non-studio conditions, while the added annotations are not an evaluation benchmark.
Problem
Accurate 3D ground truth for natural images is limited, with existing datasets often small-scale or captured in studio settings.
Method
The paper trains end-to-end 3D human-pose ConvNets with ordinal closer-farther relations between joints, optionally combined with 2D keypoints or volumetric representations.
Results
The ordinal-supervision approach achieves competitive performance with fully supervised ConvNets and new state-of-the-art results on standard benchmarks.
Takeaways & Limitations
Ordinal depth annotations can support 3D pose estimation in non-studio conditions and can be incorporated into typical ConvNet training procedures.
Takeaways & Limitations
The ordinal annotations are intended as an information source rather than a millimeter-level evaluation benchmark such as Human3.6M or HumanEva-I.
Abstract
from arXiv · showhide
Our ability to train end-to-end systems for 3D human pose estimation from single images is currently constrained by the limited availability of 3D annotations for natural images. Most datasets are captured using Motion Capture (MoCap) systems in a studio setting and it is difficult to reach the variability of 2D human pose datasets, like MPII or LSP. To alleviate the need for accurate 3D ground truth, we propose to use a weaker supervision signal provided by the ordinal depths of human joints. This information can be acquired by human annotators for a wide range of images and poses. We showcase the effectiveness and flexibility of training Convolutional Networks (ConvNets) with these ordinal relations in different settings, always achieving competitive performance with ConvNets trained with accurate 3D joint coordinates. Additionally, to demonstrate the potential of the approach, we augment the popular LSP and MPII datasets with ordinal depth annotations. This extension allows us to present quantitative and qualitative evaluation in non-studio conditions. Simultaneously, these ordinal annotations can be easily incorporated in the training procedure of typical ConvNets for 3D human pose. Through this inclusion we achieve new state-of-the-art performance for the relevant benchmarks and validate the effectiveness of ordinal depth supervision for 3D human pose.
1. Introduction
The paper addresses limited 3D annotations for natural images by training 3D human-pose ConvNets with ordinal joint-depth relations. It demonstrates competitive performance across settings, extends LSP and MPII with annotations, and reports state-of-the-art benchmark results.
- 3D human-pose datasets are limited in scale or restricted to studio settings because accurate 3D ground truth is difficult to collect.
- Human annotators can provide pairwise closer-farther joint relations across varied images, activities, viewpoints, and imaging conditions.
- The relations support direct joint-depth prediction, 2D-keypoint-plus-depth 3D prediction, and volumetric 3D-pose representations.
- Ordinal depth relations provide weaker supervision that can bypass the need for accurate 3D ground truth in end-to-end pose estimation.
- The authors report competitive performance with fully supervised approaches and new state-of-the-art results on standard benchmarks.
- LSP and MPII are augmented with ordinal depth annotations to evaluate 3D pose estimation in non-studio conditions.
2. Related work
Related work includes reconstruction methods that lift detected 2D joints to plausible 3D poses and annotation efforts using interactive tools, viewpoint labels, or pose attributes. The paper differs by providing larger-scale human ordinal-depth annotations and a simpler training scheme based on 2D keypoints and depth relations.
- Reconstruction approaches: Reconstruction methods detect 2D joints and estimate plausible 3D poses, including approaches that lift 2D keypoints with neural networks.
- 3D pose annotations: Some prior annotation efforts use interactive 3D-pose adjustment, yaw labels, or relative 3D body-part attributes.
- 3D pose annotations: The paper provides ordinal-depth annotations for more than 15k images, compared with 1k images for the posebits dataset.
- 3D pose annotations: Its training scheme requires only 2D keypoint locations and ordinal depth relations rather than an extensive set of pose attributes.
3. Technical approach
The approach trains 3D human-pose ConvNets with ordinal depth supervision, extending it from depth prediction to coordinate, volumetric, and reconstruction-based settings. It combines partial ordinal relations with 2D keypoints or volumetric representations while preserving compatibility with end-to-end training.
- 3.1. Depth prediction: The method first predicts joint depths from images using pairwise ordinal depth relations and a differentiable ranking loss.The loss enforces separation for ordered joints and equality for joints annotated at similar depths.
- 3.1. Depth prediction: The ranking loss can train on only annotated joint pairs and tolerates inconsistent relations by learning a consensus.This makes supervision usable when pairwise annotations are incomplete or ambiguous.
- 3.2. Coordinate prediction for 3D pose: Combining predicted joint depths with 2D keypoint coordinates provides weak 3D supervision for end-to-end coordinate prediction.The network adds 2N pixel-coordinate outputs and combines ranking and keypoint L2 losses as L = Lrank + λLkeyp, with λ = 100 in the experiments.
- 3.3. Volumetric prediction for 3D pose: For volumetric pose prediction, supervision is decomposed into the 2D image plane and depth dimension while the shared volume preserves 3D consistency.The 2D heatmaps use an L2 loss, while depth distributions use the ranking loss; both are obtained by sum-pooling marginalizations of the volume.
- 3.4. Integration with a reconstruction component: The proposed supervision changes the training signal rather than the network architecture, allowing existing discriminative ConvNets to be used without accurate 3D ground truth.The reconstruction component is likewise designed for integration with outputs from the coordinate or volumetric networks.
- 3.4. Integration with a reconstruction component: A reconstruction module maps predicted 2D joint locations and ordinal depths to coherent 3D pose coordinates.It is trained with projected MoCap poses and noisy depths whose majority ordinal relations are preserved, then incorporated end-to-end after the coordinate or volumetric ConvNet.
4. Empirical evaluation
The evaluation tests ordinal supervision across annotation efficiency, prediction architectures, mixed-data training, generalization under domain shift, and standard benchmarks. Ordinal annotations remain competitive with full 3D supervision and improve results across indoor and shifted-domain evaluations.
- 4.1. Datasets: The study evaluates Human3.6M, HumanEva-I, and MPI-INF-3DHP, while extending LSP and MPII with ordinal depth annotations.The evaluation includes indoor benchmarks, a dataset with indoor and outdoor conditions, and natural-image pose datasets augmented with ordinal depth.
- 4.1. Datasets: Global joint-depth orderings require roughly 17 questions per image instead of 91, making annotation approximately 5 times faster.Annotators compare joint pairs until a globally consistent ordering is inferred.
- 4.3. Ablative studies: Ordinal supervision achieves results very close to full 3D supervision across depth, coordinate, and volume regression, with a larger gap only for the strongest architecture.The volume-regression model with two hourglasses reaches average error below 80mm, where highly accurate 3D ground truth may become necessary.
- 4.3. Ablative studies: Adding ordinal depth supervision to 2D-keypoint training further improves Human3.6M performance, while reconstruction refinement provides additional gains.The complete configuration corresponds to state-of-the-art results for the dataset.
- 4.4. Comparison with state-of-the-art: The complete approach reduces relative error by over 10% on average on both Human3.6M and HumanEva-I, and outperforms prior methods across Human3.6M tables.Human3.6M gains are especially notable for actions with self-occlusions and rare poses; HumanEva-I results improve despite largely saturated numbers.
- 4.3. Ablative studies: On MPI-INF-3DHP, ordinal supervision improves generalization under domain shift, and the approach outperforms two baselines without training on that dataset.The margin is particularly large for the Outdoor sequence.
- 4.4. Comparison with state-of-the-art: Qualitative MPI-INF-3DHP results are shown from the original and a novel viewpoint.The figure provides a qualitative view of predictions under viewpoint change.
5. Summary
The paper trains end-to-end ConvNets for 3D human pose estimation using ordinal joint-depth relations instead of accurate 3D ground truth. It extends MPII and LSP with ordinal annotations and reports competitive and state-of-the-art benchmark results.
- Ordinal depth relations provide weaker supervision for training end-to-end ConvNets without accurate 3D ground truth.
- The approach works across recent ConvNet architectures, achieving competitive performance with fully supervised versions.
- MPII and LSP were extended with ordinal depth annotations, enabling evaluation under non-studio conditions.
- Incorporating ordinal annotations into recent 3D-pose ConvNets achieved state-of-the-art results on standard benchmarks.