PaperlayerView exact revision on arXiv

Source-linked AI summary

Whole-Body Human Pose Estimation in the Wild

Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, Ping Luo

arXiv:2007.11858v1cs.CV

TL;DR

Whole-body pose estimation lacks complete annotations and therefore has relied on separately trained models across face, hand, and body datasets. The paper introduces COCO-WholeBody and the single-network ZoomNet, which handles body-part scale differences through hierarchical, zoomed localization. COCO-WholeBody supports whole-body training and pretraining for facial and hand keypoint tasks, while ZoomNet significantly outperforms prior methods.

  • Problem

    Existing human pose datasets lack manual annotations for the entire body, forcing separate models trained on face, hand, and body datasets.

  • Method

    The paper introduces COCO-WholeBody and ZoomNet, a top-down single network that estimates body keypoints before zooming into face and hand regions.

  • Results

    ZoomNet significantly outperforms previous state-of-the-art methods on COCO-WholeBody, while the dataset generalizes as pretraining data for facial landmark localization and hand keypoint estimation.

  • Takeaways & Limitations

    COCO-WholeBody provides a benchmark for whole-body pose estimation and a pretraining resource for related face and hand localization tasks.

Abstract

from arXiv · show

This paper investigates the task of 2D human whole-body pose estimation, which aims to localize dense landmarks on the entire human body including face, hands, body, and feet. As existing datasets do not have whole-body annotations, previous methods have to assemble different deep models trained independently on different datasets of the human face, hand, and body, struggling with dataset biases and large model complexity. To fill in this blank, we introduce COCO-WholeBody which extends COCO dataset with whole-body annotations. To our best knowledge, it is the first benchmark that has manual annotations on the entire human body, including 133 dense landmarks with 68 on the face, 42 on hands and 23 on the body and feet. A single-network model, named ZoomNet, is devised to take into account the hierarchical structure of the full human body to solve the scale variation of different body parts of the same person. ZoomNet is able to significantly outperform existing methods on the proposed COCO-WholeBody dataset. Extensive experiments show that COCO-WholeBody not only can be used to train deep models from scratch for whole-body pose estimation but also can serve as a powerful pre-training dataset for many different tasks such as facial landmark detection and hand keypoint estimation. The dataset is publicly available at https://github.com/jin-s13/COCO-WholeBody.

1 Introduction

Whole-body pose estimation lacks a representative dataset with complete annotations, forcing fragmented multi-dataset and multi-network solutions. The paper introduces COCO-WholeBody and ZoomNet to provide unified supervision and handle scale variation across body parts.

  • Whole-body pose estimation localizes body, face, hand, and foot keypoints simultaneously for downstream applications including virtual and augmented reality.
  • Existing datasets lack manual annotations for the entire body, so prior systems train separate models on face, hand, and body datasets.OpenPose exemplifies this multi-network approach.
  • COCO-WholeBody is the first large-scale benchmark with whole-body annotations, extending COCO with face and hand boxes plus face, hand, and foot keypoints.It supports evaluation of representative methods and cross-dataset generalization.
  • ZoomNet is a top-down, end-to-end single network that estimates body structure before zooming into hand and face regions for higher-resolution localization.It integrates five network heads with shared low-level features.
  • ZoomNet significantly outperforms previous state-of-the-art methods on the whole-body pose estimation task.

2 Related Work

Related work covers separate RGB keypoint datasets for bodies, hands, and faces, alongside bottom-up and top-down localization methods. COCO-WholeBody complements these resources with large-scale, challenging in-the-wild annotations and motivates a unified top-down model.

  • 2.1 2D Keypoint Localization Dataset: COCO provides 17 body keypoints in uncontrolled conditions, whereas COCO-WholeBody adds 133 densely annotated face, hand, and foot keypoints.Whole-body estimation is harder because faces and hands require higher localization accuracy and differ in scale from bodies.
  • 2.1 2D Keypoint Localization Dataset: Existing RGB hand datasets are commonly synthetic or laboratory-captured, while COCO-WholeBody contains about 100K labeled hands in unconstrained environments.Its hand images include occlusion, interactions, motion blur, and small scales.
  • 2.1 2D Keypoint Localization Dataset: COCO-WholeBody follows 300W's 68-face-keypoint setting but is larger and includes more blurry and small-scale facial images.
  • 2.2 Keypoints Localization Method: Bottom-up methods detect and group all persons' keypoints, whereas top-down methods detect person boxes before predicting normalized poses.Top-down approaches are generally less sensitive to scale variation between human instances.
  • 2.2 Keypoints Localization Method: Earlier whole-body systems use separate models for different keypoint types, while ZoomNet integrates five components into one network with shared low-level features.The paper positions this design as a response to the training and computational costs of multiple networks.

3 COCO-WholeBody Dataset

COCO-WholeBody is presented as the first large-scale whole-body pose benchmark, combining dense annotations with statistics that expose its scale, blur, and hand-pose challenges.

  • Data Annotation: COCO-WholeBody extends COCO with face, hand, and foot annotations alongside original body labels, providing the first large-scale whole-body pose benchmark.Each person has person, face, left-hand, and right-hand boxes; invalid regions are masked during training and evaluation.
  • Data Annotation: 133 keypoints comprise 17 body, 6 foot, 68 face, and 42 hand landmarks, with face and hand boxes labeled valid only when sufficiently clear.The annotation process uses pre-trained detectors for pseudo-labels followed by manual correction, while foot keypoints are manually labeled.
  • Dataset Statistics: About 130K face and left/right hand boxes yield more than 800K hand keypoints and 4M face keypoints in total.The dataset statistics also summarize annotated keypoints and boxes for hands, face, and body.
  • Dataset Statistics: Hand and face keypoints have much smaller scales than body keypoints, creating substantial scale variation that makes simultaneous localization challenging.The scale distribution is measured using average keypoint distances in a tree-structured skeleton.
  • Dataset Statistics: Compared with 300W and Panoptic, COCO-WholeBody contains more challenging facial images and hand poses, including blurry faces and hands grasping or holding objects.Most facial images have blurriness values between 1 and 3, but WholeBody has greater variance and more images below 1.
  • Dataset Impact: COCO-WholeBody’s diversity is intended to support whole-body pose estimation as well as related face and hand keypoint tasks.Cross-dataset evaluations validate its use as a pre-training dataset for facial landmark localization and hand keypoint estimation.

4 ZoomNet: Whole-Body Pose Estimation

ZoomNet uses a top-down, single-network pipeline that first estimates body structure and coarse face-hand regions, then zooms into those regions for higher-resolution localization.

  • Pipeline: ZoomNet receives human body candidates from an off-the-shelf FasterRCNN detector and localizes whole-body keypoints for each candidate.Its pipeline predicts body/foot and face/hand keypoints successively within one network.
  • Architecture: FeatureNet extracts shared low-level features for BodyNet and the face/hand heads, supporting a unified single-network architecture.The implementation uses feature maps F1 and F2 at progressively lower resolutions.
  • Design Motivation: The top-down zooming design addresses scale variance by using coarse body localization to focus detailed prediction on smaller face and hand regions.Unlike approaches assembling multiple independently trained networks, ZoomNet is end-to-end trainable as a single network.
  • BodyNet: BodyNet jointly predicts body and foot keypoints together with face, left-hand, and right-hand bounding boxes represented by corner and center points.This produces 38 keypoints per person simultaneously.
  • Face/Hand Heads: FaceHead and HandHead crop predicted regions with RoIAlign, upscale them to higher resolution, and predict face or hand heatmaps separately.Features from F1 and F2 are resized and processed in parallel for accurate local keypoint estimation.
  • Implementation: ZoomNet uses HRNet-W32 for BodyNet and HRNetV2p-W18 for FaceHead and HandHead in the reported implementation.The architecture can also be based on other state-of-the-art network architectures.

5 Experiments

Experiments evaluate whole-body methods on COCO-WholeBody, compare single-stage and ZoomNet designs, and test cross-dataset transfer and sensitivity to bounding boxes, scale, blur, and pose.

  • Evaluation on COCO-WholeBody Dataset: Directly predicting all 133 keypoints reduces body accuracy substantially for one-stage methods, whereas ZoomNet’s decoupled design limits the decrease to 1.5% mAP and 0.7% mAR.The corresponding decreases are 14.3% and 14.2% for PAF, 17.7% and 17.0% for AE, and 9.9% and 10.0% for HRNet.
  • Evaluation on COCO-WholeBody Dataset: 10.9% mAP and 13.8% mAR separate ZoomNet from HRNet, demonstrating the reported benefit of the zoom-in design for scale variation.HRNet is treated as a one-stage alternative sharing the HRNet-W32 backbone.
  • Evaluation on COCO-WholeBody Dataset: 27.36G Flops gives ZoomNet lower complexity than OpenPose’s 451.09G and SN’s 272.30G, while its runtime averages 174.7ms/image including detection.SN averages 215.5ms/image on one GTX-1080 GPU.
  • Cross-dataset Evaluation: Pretraining on WBH improves EPE on Panoptic by 6.5%, from 7.49 to 7.00.Training and testing on WBH yields 6.66 NME and 2.76 EPE, compared with 0.68 NME and 7.49 EPE on Panoptic.
  • Analysis: Oracle face and hand boxes improve face, hand, and whole-body mAP by 19.6%, 8.4%, and 23.6%, respectively.The analysis replaces predicted boxes with ground-truth boxes before rerunning the face and hand heads.
  • Analysis: Bottom-up methods perform worse at medium than large person scales, while top-down HRNet and ZoomNet show higher medium-scale mAP.The passage attributes this contrast to bottom-up sensitivity to scale variance and difficulty detecting smaller people.
  • Analysis: Facial landmark performance is strongly affected by blur, with reported values of 2.51 versus 19.13, but is more robust to face pose, at 9.02 versus 13.77.The experiments vary image blurriness and yaw angles on WholeBody-Face.
  • Analysis: Hand-pose evaluation finds palm and varied gestures more challenging than fists on WholeBody-Hand.The comparison uses NME across fist, palm, and other hand poses.

6 Conclusion

The paper concludes that COCO-WholeBody provides a large-scale benchmark for whole-body pose estimation, while ZoomNet addresses extreme scale differences through focused face and hand localization.

  • Conclusion: COCO-WholeBody is presented as the first large-scale benchmark for whole-body human pose estimation.The paper also reports extensive evaluation of existing approaches on this dataset.
  • Conclusion: Cross-dataset evaluation demonstrates the dataset’s generalization ability.The conclusion links this result to the proposed dataset’s broader utility.
  • Conclusion: ZoomNet addresses extreme scale differences among body parts by focusing more attention on difficult-to-detect face and hand keypoints.The paper reports that experiments show ZoomNet significantly outperforms prior methods.
  • Conclusion: Face and hand keypoints use semi-automatic annotation, while foot keypoints are labeled directly by hand.Face and hand boxes are manually annotated, models provide pre-annotations, and annotators correct the keypoints.
  • Conclusion: The annotation defines 68 face joints, 21-joint hand models, and six foot keypoints with specified ordering and inner-center placement.Foot labels cover both big toes, small toes, and heels.

B Baseline Implementation Details

Baseline implementations adapt established bottom-up, top-down, multi-network, and single-network pose estimators to COCO-WholeBody using official codes and fair testing settings.

  • Baseline Implementation Details: Existing methods are reproduced with official codes, matched training parameters, and single-scale testing for fair comparison.The reimplementations are also checked against original 17-keypoint COCO results.
  • Baseline Implementation Details: OpenPose separately trains body, face, and hand models as a multi-network whole-body system.Its body, facial landmark, and hand pose components are trained on COCO-WholeBody independently.
  • Baseline Implementation Details: The single-network baseline predicts 133 keypoints and uses 134 PAFs arranged in a tree structure, with face, hand, and foot points connected to nearby body points.The implementation follows three PAF stages and one confidence-map stage.
  • Baseline Implementation Details: PAF affinity directions are calculated at image scale before down-sampling, preventing tiny-hand and tiny-face predictions from being affected by down-sampling.The method uses CPM as its backbone in the described implementation.
  • Baseline Implementation Details: Associative Embedding uses a four-stacked hourglass backbone and follows the original training settings.The approach groups keypoints through associative embeddings.
  • Baseline Implementation Details: ZoomNet outperforms SN significantly on the COCO-foot benchmark in body-foot AP.Table 6 reports the comparison, with some SN results copied from prior work.
  • Baseline Implementation Details: HRNet is adapted by increasing its output to 133 keypoints and using HRNet-W32, serving as the single-stage alternative to ZoomNet.The comparison is intended to assess the multi-stage localization design.

C ZoomNet Implementation Details

ZoomNet trains heatmap-based keypoint heads with shared loss weighting and combines separate pretraining with end-to-end fine-tuning.

  • ZoomNet Implementation Details: 2D Gaussian confidence heatmaps with σ = 3 encode keypoint locations, and summed SSE losses train corner and body keypoints.Body, face, hand, and foot losses receive equal weights.
  • ZoomNet Implementation Details: BodyNet and FaceHead/HandHead are pretrained separately before end-to-end fine-tuning for 120 epochs.Training uses random scaling, rotation, and flipping; Adam starts at 1e-3 and decays at epochs 80 and 100.

Experiments on Foot Keypoint Dataset

ZoomNet is evaluated on the COCO-foot dataset and significantly outperforms Single-Network. The section also references Table 7’s joint-learning evaluation.

  • ZoomNet significantly outperforms Single-Network on the COCO-foot dataset.The evaluation directly compares ZoomNet with OpenPose and Single-Network.
  • Table 7 is titled “Effectiveness of joint learning.”

D.1 Experiments about joint learning.

The joint-learning study compares joint training, feature reuse, and independent detectors, while the surrounding discussion emphasizes the multi-level structure of body, face, and hands.

  • Joint training improves face performance from 0.609 to 0.623 and hand performance from 0.393 to 0.401 over reusing features.The comparison is reported for ZoomNet’s BodyNet, FaceHead, and HandHead.
  • Fully independent face and hand detectors require two additional models but provide only limited gain, improving from 0.541 to 0.543.
  • Face and hand detection is more challenging than body detection because these parts occupy relatively small image regions.
  • The error analysis defines Jitter, Miss, Inversion, and Swap as distinct localization-error categories.

Error Analysis

ZoomNet’s error analysis identifies Miss as the dominant localization error and shows that hand keypoints remain harder than body and face keypoints. Error impacts vary with person size, especially for hands.

  • ZoomNet is reported to outperform Faster R-CNN in face and hand bounding-box detection through its multi-level design.The design is described as better handling scale variance.
  • ZoomNet significantly outperforms Single-Network in localization-error distributions for body, face, hand, and whole-body.
  • Miss is the major localization error for body, face, hand, and whole-body predictions.The analysis also distinguishes Jitter, Inversion, Swap, and Good predictions.
  • Hand keypoint accuracy is lower than body and face accuracy, while ZoomNet has a higher proportion of Good keypoints than Single-Network.
  • The analysis groups person sizes into medium, large, extra-large, and extra-extra large using person area.The groups are denoted M, L, XL, and XX.
  • Figure 11 reports AP improvement after correcting Miss, Swap, Inversion, and Jitter across body, face, and hand instances.Dashed red lines indicate improvement across all instance sizes.
  • Hand-pose errors affect larger instances more, where accurate localization is required despite motion blur and severe occlusion.

Qualitative Analysis

Qualitative evaluations test ZoomNet under occlusion, close proximity, and small-scale-person challenges, and compare it with OpenPose and Single-Network. The comparisons emphasize face and hand localization and scale robustness.

  • Figures 12 and 13 evaluate ZoomNet under occlusion, close proximity, and small-scale-person conditions.
  • ZoomNet qualitatively outperforms OpenPose and Single-Network, especially on face and hand keypoints.
  • The qualitative comparison highlights ZoomNet’s greater robustness to scale variance.
Loading 2007.11858v1…