Source-linked AI summary
DensePose: Dense Human Pose Estimation In The Wild
Rıza Alp Güler, Natalia Neverova, Iasonas Kokkinos
TL;DR
The paper addresses dense correspondence between RGB image pixels and a 3D human-body surface. It collects COCO-DensePose annotations and trains CNN systems with inpainted supervision, region-based architectures, and cascades. The resulting models recover highly accurate correspondences in complex scenes at multiple frames per second, while performance still has room for improvement.
Problem
Dense human pose estimation lacks manually collected ground truth for real images, while existing surrogate datasets provide weaker supervision.
Method
The paper collects dense image-surface correspondences for COCO persons and trains CNN systems using sparse annotations, inpainted supervision, region-based models, and cascades.
Results
The resulting architectures recover highly accurate dense correspondences between images and human-body surfaces in complex scenes at multiple frames per second.
Takeaways & Limitations
COCO-DensePose and the proposed architectures support dense human pose estimation and may facilitate downstream augmented-reality, graphics, and semantic 3D-object understanding tasks.
Takeaways & Limitations
Model-fitting surrogates can fail under occlusions or extreme poses, while rendered-image supervision can suffer covariate shift from natural images.
Abstract
from arXiv · showhide
In this work, we establish dense correspondences between RGB image and a surface-based representation of the human body, a task we refer to as dense human pose estimation. We first gather dense correspondences for 50K persons appearing in the COCO dataset by introducing an efficient annotation pipeline. We then use our dataset to train CNN-based systems that deliver dense correspondence 'in the wild', namely in the presence of background, occlusions and scale variations. We improve our training set's effectiveness by training an 'inpainting' network that can fill in missing groundtruth values and report clear improvements with respect to the best results that would be achievable in the past. We experiment with fully-convolutional networks and region-based models and observe a superiority of the latter; we further improve accuracy through cascading, obtaining a system that delivers highly0accurate results in real time. Supplementary materials and videos are provided on the project page http://densepose.org
1. Introduction
Dense human pose estimation maps pixels in a single RGB image to a 3D surface representation of the human body, supporting understanding beyond landmark localization. The paper introduces COCO-DensePose and CNN architectures that recover accurate correspondences in complex scenes, including multiple people.
- Motivation: Dense human pose estimation establishes correspondences from image pixels to a surface-based 3D representation of the human body.The task relates to object detection, pose estimation, and part and instance segmentation, while extending beyond plain landmark localization.
- Motivation: Unlike prior depth-based approaches, this work establishes surface-to-pixel correspondences from a single RGB image.The method addresses images without an available depth sensor.
- Results: The resulting discriminatively trained system recovers highly accurate correspondence fields for complex scenes involving tens of persons at real-time speed.On a GTX 1080, it runs at 20–26 frames per second for 240 × 320 images and 4–5 frames per second for 800 × 1100 images.
- Contributions: Region-based models outperform fully convolutional networks, while cascading further improves dense correspondence accuracy.The systems regress body surface coordinates at image pixels and combine fully convolutional and Mask-RCNN-based approaches.
2. COCO-DensePose Dataset
COCO-DensePose provides manually collected image-to-surface correspondences for 50K humans through an efficient two-stage annotation pipeline. The section also introduces annotator-accuracy analysis and pointwise and per-instance evaluation measures for dense correspondence.
- 50K humans and more than 5 million manually annotated correspondences form the COCO-DensePose dataset.
- Annotation pipeline: The annotation pipeline first segments visible semantic body parts, then maps sampled image points to corresponding surface locations using UV parameterizations and multiple rendered views.Parts are designed to simplify UV parametrization, while synchronized views and ordered image points support geometrically consistent annotations.
- Annotator accuracy: Annotation errors are smaller on distinctive parts such as the face, hands, and feet, but larger on uniform, clothing-covered regions including the torso, back, and hips.
- Evaluation measures: Pointwise evaluation uses the area under the Ratio of Correct Point curve, reporting AUC10 and AUC30 for 10cm and 30cm thresholds.AUC10 measures more refined correspondence accuracy, and the measure applies to single- and multi-person scenarios.
- Evaluation measures: Per-instance evaluation introduces geodesic point similarity and reports Average Precision and Average Recall across GPS thresholds from 0.5 to 0.95.The thresholds correspond to geodesic distances from 0 to 30 cm; GPS uses κ=0.255, making a single point score 0.5 at approximately 30 cm.
3. Learning Dense Human Pose Estimation
The paper develops dense pose models that map image pixels to body-surface coordinates, comparing fully convolutional and region-based designs with cascaded refinement and interpolated supervision. Region-based processing, task interaction, and dense supervision improve robustness and accuracy while retaining real-time inference.
- Region-based architecture: DensePose-RCNN combines Mask-RCNN-style region processing with a fully convolutional branch that predicts body-part labels and continuous surface coordinates.The system extends DenseReg with region proposals, ROI feature extraction, and task-specific dense prediction heads.
- Fully-convolutional baseline: FCN prediction first classifies each pixel into a body part, then regresses its continuous U,V coordinates within that part.The model uses 25-way classification, 24 part-specific regressors, cross-entropy, and smooth L1 losses.
- Region-based architecture: Region-based models provide scale selection through ROI pooling and decompose dense pose estimation into region-adapted modules.The architecture is designed to reduce the burden on a single fully convolutional network handling segmentation, localization, and scale variation.
- Efficiency: Inference runs at 25fps on 320x240 images and 4-5fps on 800x1100 images with a GTX1080.These timings establish the reported real-time operating range for the proposed system.
- Interpolated supervision: A teacher network inpaints sparse annotations into dense supervision, retaining predictions only on human foreground regions.The inpainted signal is used to train the region-based system and improves performance over sparse points or existing datasets.
- Cascaded refinement: Cross-cascading combines DensePose predictions with masks and keypoints before refining each task branch.Cascading is intended to provide contextual information, deep supervision, and complementary task guidance.
4. Experiments
Experiments evaluate DensePose on COCO, comparing manual annotations with surrogate supervision and testing FCN, region-based, distilled, and cascaded systems. Region-based processing, inpainted supervision, and cascading improve performance in realistic multi-person scenes with scale, pose, background, and occlusion variation.
- Experimental setup: The evaluation uses 48K training humans and a test set of 1.5k images containing 2.3k humans from COCO.The test partition is COCO keypoints-minival and the training partition is COCO-train.
- Supervision comparison: Model-fitting supervision can fail under occlusion or extreme poses, producing rejected examples and potentially systematic ground-truth errors.The UP dataset is described as vulnerable to fitting failures, including frequent foot misalignment after filtering.
- Supervision comparison: Synthetic supervision can suffer covariate shift because rendered images have different statistics from natural images.The paper compares against SURREAL and UP using the shared SMPL surface model.
- Supervision comparison: DensePose supervision outperforms surrogate datasets, while combining surrogate datasets improves over either surrogate alone but remains below DensePose.The comparison uses ResNet-101 stride-8 FCNs with data augmentation and Deeplab-type architectures.
- Multi-person evaluation: FCNs degrade substantially in-the-wild because they are not sufficiently robust to COCO’s variability in object scale.Multi-scale testing does not eliminate the scale-robustness problem reported for DensePose-FCN.
- Multi-person evaluation: Switching to DensePose-RCNN gives a large performance boost, while dense distilled supervision further improves results and cascading nearly matches the privileged DensePose∗ evaluation.The privileged setting uses ground-truth boxes, background removal, fixed scale, and scale ensembling, so it acts as an approximate upper bound for in-the-wild performance.
- Qualitative results: Qualitative results handle substantial occlusion, scale, and pose variation while hallucinating body structure beneath dresses or skirts.The paper visualizes correspondences as fishnet isocontours over detected humans.
5. Conclusion
The paper introduces COCO-DensePose and discriminatively trained architectures for accurate image-to-body-surface correspondence. The resulting system operates at multiple frames per second and is presented as a basis for augmented reality, graphics, and semantic 3D understanding.
- Conclusion: The work introduces COCO-DensePose, a large-scale dataset of ground-truth image-surface correspondences, and novel architectures for accurate dense correspondence.The conclusion frames the contribution as discriminatively trained models for dense human pose estimation.
- Conclusion: The system recovers dense body-surface correspondences at multiple frames per second.This is the paper’s reported deployment-level performance claim.
- Conclusion: The authors anticipate applications in augmented reality and graphics, as well as progress toward associating images with semantic 3D object representations.These are stated as potential downstream directions rather than demonstrated general-purpose capabilities.