Source-linked AI summary
Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose
Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, Kostas Daniilidis
TL;DR
Single-image 3D human pose estimation remains challenging, while prior ConvNet methods often rely on two-step pipelines or restrictive assumptions. The paper predicts per-joint voxel likelihoods in a discretized 3D space and refines them with coarse-to-fine supervision. It achieves state-of-the-art benchmark results with a relative error reduction greater than 30% on average, while a decoupled variant supports settings without corresponding 3D groundtruth.
Problem
Single-image 3D human pose estimation is an open, ill-posed challenge, and prior approaches often rely on restrictive assumptions or additional information.
Method
The paper performs end-to-end 3D keypoint localization using per-voxel joint likelihoods in discretized space and coarse-to-fine supervision for iterative processing.
Results
More than 30% relative error reduction on average is reported for state-of-the-art results on standard benchmarks against ConvNet-only and hybrid approaches.
Takeaways & Limitations
The volumetric representation also supports a decoupled architecture for in-the-wild images when end-to-end training with corresponding 3D groundtruth is unavailable.
Takeaways & Limitations
On MPII, which has 2D annotations but no 3D groundtruth, performance cannot be quantified and only qualitative results are provided.
Abstract
from arXiv · showhide
This paper addresses the challenge of 3D human pose estimation from a single color image. Despite the general success of the end-to-end learning paradigm, top performing approaches employ a two-step solution consisting of a Convolutional Network (ConvNet) for 2D joint localization and a subsequent optimization step to recover 3D pose. In this paper, we identify the representation of 3D pose as a critical issue with current ConvNet approaches and make two important contributions towards validating the value of end-to-end learning for this task. First, we propose a fine discretization of the 3D space around the subject and train a ConvNet to predict per voxel likelihoods for each joint. This creates a natural representation for 3D pose and greatly improves performance over the direct regression of joint coordinates. Second, to further improve upon initial estimates, we employ a coarse-to-fine prediction scheme. This step addresses the large dimensionality increase and enables iterative refinement and repeated processing of the image features. The proposed approach outperforms all state-of-the-art methods on standard benchmarks achieving a relative error reduction greater than 30% on average. Additionally, we investigate using our volumetric representation in a related architecture which is suboptimal compared to our end-to-end approach, but is of practical interest, since it enables training when no image with corresponding 3D groundtruth is available, and allows us to present compelling results for in-the-wild images.
1. Introduction
Single-image 3D human pose estimation is challenging because the task is ill-posed and prior ConvNet approaches often rely on intermediate 2D localization or restrictive assumptions. The paper instead uses volumetric keypoint localization with coarse-to-fine supervision, achieving state-of-the-art benchmark results and practical performance for in-the-wild images.
- Motivation: Single-image 3D human pose estimation is an open challenge because the task is ill-posed and often requires restrictive assumptions or additional information.Prior settings include background subtraction, groundtruth 2D joints, multiple camera views, or temporal consistency.
- Prior approaches: Prior end-to-end ConvNet approaches use coordinate regression, image-pose matching, or pose classification, yet traditional two-step pipelines have outperformed them.Two-step pipelines use ConvNets for 2D joint localization before recovering 3D pose.
- Volumetric representation: The paper casts 3D pose estimation as keypoint localization in a discretized 3D space, predicting per-voxel likelihoods for each joint instead of directly regressing coordinates.The volumetric representation provides groundtruth across voxels and is described as richer than a set of world coordinates.
- Coarse-to-fine prediction: Coarse-to-fine supervision addresses the volumetric representation’s increased dimensionality by progressively increasing depth resolution and enabling iterative processing.The authors report that this scheme yields more accurate estimates after each step and outperforms naively stacking more components.
- Results and practical use: More than 30% relative error reduction on average accompanies state-of-the-art results on standard benchmarks against ConvNet-only and hybrid approaches.The approach is also used in a decoupled architecture for in-the-wild images when end-to-end training is unavailable.
2. Related work
Prior work largely used coordinate regression, classification, or two-step pipelines for single-image 3D pose, while this paper places volumetric prediction within that landscape.
- ConvNet-only methods commonly regress 3D joint coordinates, while other approaches use pose embeddings or predefined pose classes.Coordinate regression targets concatenated x, y, z joint coordinates; embedding and classification alternatives trade efficiency, accuracy, or pose flexibility.
- Two-step methods use ConvNets for 2D joint localization before optimization or body-model fitting recovers 3D pose.These pipelines had generally outperformed earlier ConvNet-only approaches.
- The proposed single-network approach provides a rich 3D volume that can support post-processing while avoiding additional refinement for state-of-the-art results.The paper reports complete volume predictions in a single forward pass lasting a few milliseconds.
3. Technical approach
The paper represents each joint with voxel-wise 3D likelihoods and refines predictions coarse-to-fine to manage dimensionality. It also studies a decoupled variant for settings without paired image and 3D ground truth.
- 3.1. Volumetric representation for 3D human pose: Coordinate regression predicts a 3N-dimensional vector of joint coordinates but makes the learning problem highly non-linear and permits hierarchical error propagation.The coordinates may be expressed globally relative to a root joint or locally relative to a parent joint.
- 3.1. Volumetric representation for 3D human pose: The joint location is selected from the voxel with the maximum response in each predicted 3D grid.The network organizes the theoretical w×h×d×N output as w×h×dN channels in practice.
- 3.1. Volumetric representation for 3D human pose: The volumetric representation discretizes 3D space and predicts per-voxel likelihoods for each joint instead of committing immediately to one coordinate.Each joint receives a w×h×d volume supervised by a 3D Gaussian centered at its ground-truth position.
- 3.2. Coarse-to-fine prediction: Iterative processing and intermediate supervision refine predictions and provide richer gradient signals across multiple network stages.These design choices are motivated by their effectiveness in 2D human pose estimation.
- 3.2. Coarse-to-fine prediction: Coarse-to-fine supervision gradually increases depth resolution because naive stacking is hindered by the representation’s large dimensionality.At 64 × 64 × 64 resolution with 16 joints, the network would estimate likelihoods for more than 4 million voxels.
- 3.3. Decoupled architecture with volumetric target: The decoupled variant predicts 2D keypoint heatmaps first, then infers 3D joint positions with the volumetric representation.Its two stages can use separate, non-corresponding data sources when paired image and 3D ground truth are unavailable.
- 3.3. Decoupled architecture with volumetric target: The decoupled training strategy is practical for in-the-wild images but remains suboptimal compared with end-to-end training when paired 3D ground truth is available.
4. Empirical evaluation
The evaluation tests volumetric pose prediction and its coarse-to-fine refinement across standard benchmarks, using dataset-specific protocols and metrics. The approach improves over coordinate regression, naive stacking, and prior methods, while also producing qualitative in-the-wild results.
- Datasets: The evaluation covers Human3.6M, HumanEva-I, KTH Football II, and qualitative MPII results, with direct image-to-3D training unavailable or impractical for MPII and KTH.Human3.6M and HumanEva-I provide quantitative evaluation, while MPII has no 3D groundtruth and is evaluated qualitatively.
- Evaluation setup: The volumetric representation predicts image coordinates and metric depth relative to the root using a discretized x-y grid and z grid in [−1, 1] meters.The x-y grid lies within the image bounding box, while the z grid is centered at the root joint.
- Component evaluation: 112.41mm coordinate-regression error establishes the Human3.6M comparison point for the simpler one-hourglass component analysis.The coordinate-regression architecture uses a fully connected output layer, whereas volumetric prediction is fully convolutional.
- Component evaluation: 69.77mm with two coarse-to-fine hourglasses beats 75.06mm from naive stacking with four hourglasses, using less than half as many parameters.Naive stacking provides diminishing returns, whereas coarse-to-fine supervision improves performance with two, three, or four hourglasses.
- Comparison with state-of-the-art: The network outperforms other methods on average on Human3.6M and all published approaches on HumanEva-I, while also outperforming single-view methods on KTH Football II.On Human3.6M it also achieves large margins under reconstruction error, and on KTH it is competitive with multi-view results.
5. Summary
The paper presents volumetric 3D pose estimation with coarse-to-fine supervision and reports strong benchmark performance, alongside qualitative results across several datasets.
- The approach achieves a relative error reduction greater than 30% on average on standard benchmarks.
- Human3.6M results are quantitatively compared using average 3D joint error in millimeters.
- Qualitative results include original and novel viewpoints for Human3.6M, HumanEva, KTH Football II, and MPII.